There are two traditional tests of a business's insolvency; the 'Balance Sheet Test' which is failed if the business's liabilities are higher than its assets and the 'Cash Flow Test' which is failed if the business cannot pay all of its creditors, in full, as and when they fall due.
Whilst these two tests alone do not prove certain insolvency, they are strong indicators and you should consider seeking professional advice if your business fails either one.
Other warning signs to look out for include:
Whilst this is not an exhaustive list the above are all strong indicators of financial distress and if you are experiencing any of the above you should consider taking professional advice.
The earlier you seek advice, the more that can be done to save your business.