Preheat the oven to 350 degrees. Prepare an 8x8 pan by spraying it with oil and lining it with parchment paper. In a medium mixing bowl, whisk together the flour, baking soda, baking powder, salt, cinnamon, ginger, and nutmeg.
In a separate larger bowl, whisk together the oil, brown sugar, granulated sugar, eggs, buttermilk, vanilla, and carrots.
Add the dry ingredients to the wet. Fold until just combined.
Pour the batter into the prepared pan.
Bake for 34-37 minutes, or until a toothpick inserted in the center comes out mostly clean. There should not be any raw batter. Remove from the oven and cool on a wire rack for 30 minutes before removing.
Place the cake in the refrigerator until it cools to room temperature or leave it on the counter for a few hours to cool at room temperature. While it cools, in a large mixing bowl with a hand mixer (or in a stand mixer fitted with the paddle attachment), beat the cream cheese on high until light and fluffy. This will take 2-3 minutes.
Add the butter and beat on high until well combined.
Add the confectioners sugar and vanilla. Mix on the low until the sugar is absorbed. Increase the speed to high and mix for another 1-2 minutes until creamy.
Once the cake has cooled, spoon all of the frosting on top of it and spread into an even layer using an offset spatula, a spoon, or a butter knife. Garnish with chopped nuts if desired.