Every time I have played with an orchestra, it is an amazing experience. It doesn't matter if the orchestra is great or if it is terrible. For the pianist, being in such a huge group setting is usually a rare thing. One of the things I love most about concerto playing is the way the sound of the piano constantly evolves as it is supplemented with all the different timbres in the orchestra. It is like playing a whole new instrument.
For those of us that love chamber music, the first Brahms concerto is great. The level of communication and prior discussion that must exist between the conductor and the soloist is very high. The piano and the orchestra constantly complement each other, but the writing is not so dense as to exclude the use of a rubato normally reserved for chamber playing.
I've found that as you add instruments to an ensemble, the liberties you are allowed to take with the music are gradually lessened, since one must take into account more musical lines, more individuals and more opinions. I haven't seen this to be the case with the Brahms concerto. The conductor and the piano form a duo in which both participants must interact constantly. It acts as a chamber group that constantly changes. At times it is a horn and piano duo, at times a piano quintet, a string quartet, a wind quintet, a clarinet duo plus piano or an oboe duo plus piano; there are many combinations. Even so, it never sounds "small". It is a big concerto, with very loud and dense tutti sections. Since it was originally written as a symphony, the orchestra's role is just as important as the piano's and the constant interaction between the two makes for a great piece of music.
The attitude the soloist must have towards the concerto before playing is something to think about. Some pianists try to work with the orchestra, others just want to be accompanied with something loud in the background that plays tutti while they rest. And some, like the genius pianist Sviatoslav Richter, take a completely different approach, as is seen in this video of Tchaikovsky's first piano concerto:
Maybe my wife wants to be Rita at my next concert?