Heat 1 tablespoon oil in heavy large saucepan over medium-high heat. Add onion, celery, carrot and garlic; sauté until brown, about 10 minutes. Add tomato paste and cook 2 minutes. Add white wine and simmer until liquid is reduced by half, scraping up browned bits, about 5 minutes. Add next 7 ingredients; simmer until mixture is reduced to 3 cups, about 50 minutes. Discard bay leaf. Season sauce with salt and pepper.
Preheat oven to 350°F. Sprinkle halibut with salt, pepper and cayenne pepper. Heat remaining 1 tablespoon oil in heavy large skillet over high heat. Add fish to skillet and cook until brown, about 2 minutes per side. Transfer fish to plate. Pour off oil from skillet. Reduce heat to low. Add balsamic vinegar to skillet; simmer 1 minute. Add sauce. Whisk in butter. Season with salt and pepper. Pour sauce into 13x9x2-inch glass baking dish. Place fish atop sauce in dish. Bake until fish is opaque in center, about 10 minutes.
Cook potatoes and garlic in large saucepan of boiling salted water until tender, about 25 minutes. Drain well. Return potatoes and garlic to pot.
Using handheld electric mixer, beat potato mixture at low speed until smooth. Add butter; beat until melted and smooth. Increase speed and whip potatoes just until light and fluffy. Stir in herbs. Season with salt and pepper.
Divide potatoes among 6 plates. Top potatoes with fish and sauce and serve immediately.