Setting up a business today is also laying its technological foundation - the decisions made at the start will project your comfort level later on. It may seem like a small thing, but choosing a domain, hosting (shared hosting is often enough to start with, then you can switch to VPS) and a simple, well-made website really matter.
A company email is more than a nice footer header; it builds customer trust and order in correspondence. It's also a good idea to put in place basic team communication tools right away - such as a shared calendar, shared documents and instant messaging - to make daily work easier.
Management systems are key: an invoice program, a simple CRM for contacts (sometimes a well-organized spreadsheet is enough to get you started), and secure file storage in the cloud. Each of these decisions affects efficiency - and how much work it will be to migrate if you change tools later.
Security is often overlooked, and that's a mistake; backups (local and in the cloud), strong passwords, two-step verification and basic antivirus are all viable protections. You probably don't need to buy the most expensive solutions right away, but it's worth planning scalably - what works for 2-3 people may not be enough when your team grows.
In my experience: a reasonably thought-out IT infrastructure at the start is an investment that usually pays off faster than you think.