First read the contract you signed with the contracting company.. you may find that you will be in breach if you go into the same business within x months of leaving the current contracting company.

Then read it again to see what notice you have to give them and what you signed up for in the way of giving them a commission based on your pay rate. It has all got to come down to whether you can be sued or not by the contracting company.

Yes I don't use an agency to find my own work, but the rules are a bit different in the UK, so my advice would be of little use to you.

To get the advice you need, I suggest you post the country you are in and state if applicable - it would be a good idea to update your profile in the User CP to add your location.