Some 53.1 percent of Danes have rejected the adoption of EU legislation on justice and home affairs the centre-right government said would help strengthen cross-border policing.
In the country's seventh referendum on EU integration, wary citizens have once again defended national sovereignty.
Denmark may now be forced to leave Europol, the EU's law enforcement agency, which deals with organised crime, trafficking and terrorism.
Its legal status is expected to change in 2016. If this happens, Copenhagen's 1993 decision to opt out of several key EU policy areas (including justice and home affairs),means it will be unable to stay in the agency.
A win for the 'yes' camp would have meant opting in to some of these policies and would, therefore, have enabled Denmark to maintain its Europol membership.
Attacks on European capitals
The referendum comes nearly ten months after shootings at an exhibition and outside a synagogue in Copenhagen, a matter of weeks after the Paris attacks and as Europe struggles to cope with record numbers of migrants.
Opponents of a 'yes' vote argued it would result in the EU eventually dictating Denmark's immigration policies. While a 'yes' result would have given the Danish parliament the power to adopt further EU justice legislation without consulting the electorate, the government said it would hold a referendum to decide any potential changes to immigration laws.
Denmark is up there with some of Europe's strictest immigration policies. Between January and the end of November, 2015, it had received far fewer asylum seekers than neighbouring Sweden or Germany.