In a medium bowl, whisk together 2 cups flour, ½ teaspoon baking powder, ½ teaspoon baking soda and ¼ teaspoon salt until well blended; set aside.
In a mixing bowl, beat together ¾ cup butter and 1 ¼ cup granulated sugar at medium speed until light and fluffy, 4 to 5 minutes. Beat in 2 eggs and 1 egg yolk one at a time, beating well after each addition and scraping down the sides of the bowl as necessary. Beat in 1 teaspoon vanilla and ¼ teaspoon almond extract. At low speed, beat in the flour mixture in three additions, alternating it with the buttermilk in two additions.
Scrape half of the batter into the prepared pan and smooth the top with a small offset metal spatula. Sprinkle half of the crumb topping evenly over the batter, breaking up any large lumps with your fingers. Arrange a circle of peach slices, overlapping them slightly, around the outer edge of the pan. Arrange another circle of the remaining slices in the center. Spoon the remaining batter over the filling and smooth it into an even layer, covering the filling. Sprinkle the remaining crumb mixture on top of the cake.
Bake the cake for 50 to 60 minutes, until a toothpick inserted into the center comes out clean. (Cover the cake with foil during the last 15 minutes of baking if it is getting too brown.) Cool the cake in the pan on a wire rack for 20 minutes.
Remove the side of the pan and cool the cake completely.