IETF: GDPR compliance means caring about what's in your logfiles Sysadmins: while you're busy getting ready for the GDPR-regulated world, don't forget what your servers are storing in their logfiles. That advice comes courtesy of a draft mulled by the Internet Engineering Task Force's Internet Area Working Group (IETF's INTAREA). The document, here, offered a handy checklist as a set of …
" Full IP addresses should only be stored for as long as needed to provide a service;
Logs should only include the first two octets of IPv4 addresses, or first three octets of IPv6 addresses;
Inbound IP address logs shouldn't last longer than three days;
Unnecessary identifiers should not be logged – these include source port number, timestamps, transport protocol numbers, and destination port numbers;"
I don't agree. The way the internet works means that ip addresses are a necessary use. Yes, IP addresses can be Personally Identifiable Information when combined with other data, or you are using a fixed IP at an individual address, but if you access my services I can't help but know your IP address. My logging is fine to record your entire IP address. It is what I then do with that information that is important.
Also, I am bound to provide suitable protection against any intrusion, or notify ICO if I suspect an intrusion. This aso means potentially sifting through logs to try and locate that source. Three days? That is just silly. 6 Months, sensible. 12? Maybe they have a point, unless regulatory requirements state otherwise.
This would come under legitimate interest. If you come to use my online services, then I have to store the above information to allow me to satisfy the requirements that come from operating online services in the EU. If I then decide to do something funky with that data, then that is another thing entirely.
I am wondering if INTAREA felt that they hadn't yet made any statement regarding GDPR and rolled out the first thing that sounded press friendly. They certainly are not showing a deep understanding of the issues involved.
"Logs should be protected against unauthorised access."
And remember, Kids, don't take sweets from Strangers...
"Yes, IP addresses can be Personally Identifiable Information when combined with "
To be pedantic, logs containing IP addresses *are* personal data *because* they can be combined with other information likely to...
"Also, I am bound to provide suitable protection against any intrusion, or notify ICO if I suspect an intrusion. This aso means potentially sifting through logs to try and locate that source. Three days? That is just silly. 6 Months, sensible. 12? "
This is also the view of our lawyers, three days is silly. You can't possibly detect and investigate suspected breaches in three days. There were other reasons too, which I'm not allowed to mention because it would break legal privilege.
"You can't possibly detect and investigate suspected breaches in three days."
Correct! That's the point. If you can't detect a breach it never happened and you do not have to disclose it. The GDPR lawyers actually were brilliant.
Yes, there's a couple of long game attack vectors where you need to track IP addresses for longer than 3 days. However, I suspect that if you consider that you still require the full IP address for security purposes, its not a violation. It really depends on how they worded the actual rule/regulation.
But to your point... if you show that a company allowed or didn't discover breaches because of the 3 day suggestion, you will see a queue of class action attorneys getting ready to sue the company.
Now is the time for all senior IT guys to go back to school and get a law degree so that they can specialize IT legal compliance.
Don't know how it would work in the UK side of the pond, but it could be a great alternative to becoming a patent attorney.
The two parts of GDPR that apply here are:
1. You cannot store more personal information than is necessary to run your business
2. Once the data is no longer needed, it should be deleted.
If you have additional regulations, like ISPs, that says that the information must be kept for an additional time period, that is one thing.
But a normal website owner should have no further need for the data after it has been in the logs long enough to check for unauthorized access, which should be same-day or next-day (3 days if there is a weekend between), is what I'm reading from the IETF. But that does seem rather short. A few weeks seems more reasonable.
Obviously it means that you need to be pro-active in ensuring your website is secure, not waiting until the worst happens and then going back through months' worth of logs.
"But a normal website owner should have no further need for the data after it has been in the logs long enough to check for unauthorized access, which should be same-day or next-day (3 days if there is a weekend between), is what I'm reading from the IETF. But that does seem rather short. A few weeks seems more reasonable."
You won't necessarily know about an instrusion until Troy Hunt mentions your domain. Bad things™ happen even to those that do take precautions. Ever hear of the rogue employee? And you need to find out what occurred so you know that particular hole is shut down and the ICO will want to know what you are doing about the data breach. You can't do that if you dispose of your logs too quickly. When you are aware of it, you don't know how or when it occurred yet so you need to check.
Those that think they are that secure that they can't be hacked in anyway are, for the most part, deluding themselves. You have to assumed you will be hacked at some point.
“The processing of personal data to the extent strictly necessary and proportionate for the purposes of ensuring network and information security, i.e. the ability of a network or an information system to resist, at a given level of confidence, accidental events or unlawful or malicious actions that compromise the availability, authenticity, integrity and confidentiality of stored or transmitted personal data, and the security of the related services offered by, or accessible via, those networks and systems, […] by providers of electronic communications networks and services and by providers of security technologies and services, constitutes a legitimate interest of the data controller concerned. This could, for example, include preventing unauthorised access to electronic communications networks and malicious code distribution and stopping ‘denial of service’ attacks and damage to computer and electronic communication systems.”
https://gdpr-info.eu/recitals/no-49/
So a few weeks for logs? Fine. Do it. You may never need them beyond that. But if you do need to know what happend a couple of months ago?
So a few weeks for logs? Fine. Do it. You may never need them beyond that. But if you do need to know what happend a couple of months ago?
You might if you're not keeping up with the logs. Look at the breeches that went on for months without being noticed.
Then again, if you're mining user data, etc.... all bets are off on how long it's kept. I'm looking at you Google and Facebook as prime examples.
In relation to GDPR and retention, you don't do what ISPs tell you because they tell you. They do not have the authority to overrule the law.
You can store data for as long as you want, so long as you can justify it, but the principle is you should delete it when you no longer need it to do the job you are doing with it.
If you have a legal obligation to store data for say tax purposes for 6 years, then even though your relationship with the data subject to whom that data belongs comes to an end, then you can continue to store it for the full 6 years claiming it is being retained for legal reasons. If the data subject contacts you and requests the data be deleted, you can refuse citing you have a legal reason to retain it.
But an ISP cannot order you to keep the data for a certain amount of time. An ISP ordering you to keep data does not absolve you from the effect of the law if a data subject makes a complaint to the ICO.
Some of the more pedantic amongst us would ask: Why are you storing logs on an Internet facing server anyway? Anyone with more than half a clue would be storing them centrally.
Personally I would say that three days is reasonable; _IF_ you do not have a defined policy in place specifying how long you keep non-anonymised logs for, and for what purpose you are keeping them for, and a justification as to why that's reasonable.
Of course IANL.
Two words: "Shared Hosting"
What if my local shop website is using Fasthosts, or even a US-based hosting company, and is sharing Apache2 server logs with US-based sites. Do I have a requirement to demand that any access.log records pertaining to my site are restricted?
Another two words: "Managed Hosting"
Who is responsible for ensuring that the web server logs are maintained in accordance with GDPR? The hosting provider in the USA (Rackspace, for example) or the clients themselves?
This has all the makings of the same SNAFU as the Great 2012 EU Cookie Compliance Debacle.
"Do I have a requirement to demand that any access.log records pertaining to my site are restricted?"
Yes, you are the data controller..
"Who is responsible for ensuring that the web server logs are maintained in accordance with GDPR?"
The client in the first place, they will be the data controller..
Your hosting company is processing the data so they have responsibilities but the data controller determines the manner in which data is processed.
In both cases the data controller needs to ensure that this is in the contract.
Doctor Syntax is entirely correct.
You have to first identify who is the data controller and data processor in a data relationship.
Then as data controller you have to write into your contracts with your data processors GDPR terms. So you have to tell them what you expect of them.
And additionally, it is not enough to take their word that they are complying with GDPR, you as DC have to check they are. Audits might be necessary.
>>two words: "Shared Hosting"
>>"What if my local shop website is using Fasthosts, or even a US-based hosting company, and is >>sharing Apache2 server logs with US-based sites. Do I have a requirement to demand that any >>access.log records pertaining to my site are restricted?"
Simple answer - Yes you have a requirement. If you chose a hosting provide who doesn't meet your requirement that's your issue.
>>"Another two words: "Managed Hosting"
>>"Who is responsible for ensuring that the web server logs are maintained in accordance with GDPR? >>The hosting provider in the USA (Rackspace, for example) or the clients themselves?"
Again another simple answer - You do, if your provider states they're compliant with GDPR, and you've carried out appropriate due diligence, then you should be covered.
>>This has all the makings of the same SNAFU as the Great 2012 EU Cookie Compliance Debacle.
Not really, though there are some gray areas and clarifications needed GDPR is a lot better than a lot of stuff coming out of Government/EU.
GDPR does not restrict the type of data written into log files.
The question is around the type of information you are writing into a log file and whether that is considered to be personally identifiable information.
You might adopt a strategy of not storing any PII, and if you can achieve that, then you don't need to comply with GDPR.
Once you store a single item of PII then you have to comply.
A full IP address of a piece of equipment belonging to a natural living person, which enables that person to be identified is considered to be PII.
You should continue to store as much as you need in a log file to enable that log file to do its job of providing you with sufficient information for you to debug a problem.
The statement about not storing port numbers is utter nonsense: port numbers cannot be used to identify a living person.
@m0rt - "Three days? That is just silly. 6 Months, sensible. 12?"
Think you hit the nail on the head. This is just sound-byte and headline porn, but technical BS!
PCI-DSS states you should keep at least 3 months worth in a warm auditable environment, and 12 months cold stored. I've not bothered checking NIST.
You could have log level settings to include or exclude certain sensitive information, but you still have to somehow redact that sensitive information later from older logs, and logs are only supposed to be written once.
Redacting might also be classified as destroying evidence.
"Redacting might also be classified as destroying evidence."
How so?
As I understand it, it's only "destroying evidence" in a legally prosecutable sense if you know, or should know, that the data you're destroying is relevant or likely to be relevant to an active criminal investigation. Routine redactions for privacy purposes wouldn't qualify.
But I'm no lawyer, so I may very well be wrong.
Kinda sorta...
You have both Criminal and Civil statutes and then there's the argument that if you failed to protect your data or systems from a criminal hack, you're liable. So that's going to run counter and you would have the right to retain certain data.
I guess it gets down to how the law was written. If it was written poorly, it could be challenged over this and it could be overturned and they would be forced to rewrite it.
Given the level of understanding and quality of questioning we saw from US law makers when failing spectacularly to bring Facebook to heel, I'm not too worried about this.
"So, why do you keep logs on your computing device?. Do you burn them to provide sustainable energy to run it? Or is it more to hold the device down in case it gets windy?"
"No sir, it's so we can track visitors."
"Ah I see, so each visitor leaves a stick or a log as a kind of thank you gift. Very good. By the way, my grandson has a computing device. Do you think he would be pleased if I left a log on it?"
"Yes sir, I'm sure he would."
"Thank you. You are free to leave".
"Full IP addresses should only be stored for as long as needed to provide a service"
However long your intrusion detection policy needs, this is what this is for.
If you need to promulgate IP addresses beyond this point, for marketing, SEO, CMS campaign management, etc, then the redacted form is more than sufficient.
"Full IP addresses should only be stored for as long as needed to provide a service"
How is "your intrusion detection policy" a "service" (that you are providing)?
It's an internal service that your operations team provide to maintain the integrity and availability of the actual, saleable product/service that your business provides.
One is (in part) dependent on the provision of the other - your IDS is part and parcel of providing your product.
"It isn't sufficient. You will need the data subject's explicit consent."
You *may* need the subject's consent, depending on the processing purpose.
In any case, you will not need the subject's consent for handling partially redacted IPs because, assuming certain other fields (like time) are fuzzed/redacted, these cannot be reasonably considered PII. Similar applies to aggregated data. If you can't extract an individual's identity it isn't PII.
However you will have to make exactly clear that you are doing this in your privacy policy and/or terms of use.
"If you need to promulgate IP addresses beyond this point, for marketing, SEO, CMS campaign management, etc, then the redacted form is more than sufficient."
It isn't sufficient. You will need the data subject's explicit consent.
I could well be wrong here, IANAL etc., but my understanding is that fully de-identified data is not covered. E.g. if I run a shoe shop and record numbers of types of shoes sold to men and women in a tally then that aggregated data is not subject to GDPR, and having bought shoes from me you can't request that I remove the record that I sold a pair of converse to a man or a woman (delete as appropriate), or are entitled to know that you're included in that tally so long as I don't have the identifying data tying you to it.
Now, if I have some record that ties that particular sale to you I have to keep it so long as I need to fulfil any legal obligations for me to keep that information (not sure there are any pertaining to shoes), or as long as necessary to carry out processing you consented to, and you can request the data be removed (and I might have to say I'm required to retain it to comply with the 1972 Shoe Licensing Act). Where things can get sticky is if the data is granular enough to be potentially identifiable, e.g. I sell medical devices and record the customer's full postcode.
You are pretty much correct.
But I think summarising the GDPR in a couple of paragraphs like that is too simplistic. There are a set of principles and data subject rights that need to be adhered. A full description of those cannot be provided in a couple of paragraphs. You have covered in your text one right and one principle only.
Personally identifiable information is any information from which a living individual can be identified.
-> GDPR doesn't cover any data on dead people.
Things like IP addresses, names, addresses, email addresses which contain a person's name from which they can be identified, even if a business related email address), post codes, medical information, political affiliations.
You raise an interesting point in relation to aggregation. The key question is this: Can a person be identified from the data (whatever that data is, aggregated or not). The answer might be no, but the the next question is, if this dataset is combined with another at some point in the future, can the person then be identified? If the answer is yes, and that aggregation of datasets occurs and you haven't taken adequate steps to protect that data from a breach, then you are at risk of being fined under GDPR.
> Vulture South also notes that legally-mandated logging, such as to comply with local telecommunications data retention laws, isn't covered by the draft.
Wonder how this is going to play not with just data retention laws but legally mandated compliance/records keeping requirements.
For example, I'm pretty sure that patent offices around the world would need to keep logs for at least currently valid patents. They'd need to be able to answer questions like "What was the date, time, and IP address from which patent application XYZ was lodged 15 years ago?" in case of disputes or fraud.
Or "Successful visa applicant XYZ from 9 years ago was incorrectly granted a visa because we have now discovered they were in fact a wanted criminal who managed to hide their identity. We suspect that there is an identity-fraud organisation out there that specialises in hiding criminals true identities and obtains identity documents and visas for these people. So, what IP address(es) did the application for XYZ come from (so they might be able to identify information about this crime ring), and what other applications have we ever received from that same IP address (in case they submitted visas for multiple undetected criminals from the same source), and what other communications of any type have we had from that IP address?"
These requests, while made up using random requirements and organisations, are not too far off the type of requests I have had to fulfill in various positions.
Legally mandated requirements are that. Legal requirements. So if you run a Telco, you have to comply with the the data logging requirements for running that Telco.
After that GDPR and the ePrivacy directive take hold.
SO if you are legally required to keep a record of what phone calls where made through your system for 7 years, then you keep them for 7 years. But on the first day of the 8th year, you better have your data deletion policies in place.
So in the State of Nevada where the government wrote PCI into law, meaning you are obligated to comply with all provisions of the PCI DSS, it's OK to keep all of that data. Presuming you are subject to GDPR, of course.
Perhaps this could inspire multinationals to incorporate in Nevada instead of Delaware and move all of the headquarters to Las Vegas. Their travel expenses to junkets also would be reduced. Win-win!
"So in the State of Nevada where the government wrote PCI into law, meaning you are obligated to comply with all provisions of the PCI DSS, it's OK to keep all of that data. Presuming you are subject to GDPR, of course."
Yes, but if the data subject is in the EU then GDPR restricts what you can do with them. You can use them for compliance. You can't sell them on. You can't use them for marketing pestering.
We get requests from the Policy on tracking IP addresses used by customers up to 12-18 months after the bad deed happened. And most break-ins are not discovered for the first 200 days.
So we have decided, with lawyers, that we have a lawfull purpose to keep the logs for 24 months.
3 days is nothing. Often requests are referring to things that happened in the past. 30 days, and we will not be able to handle half the requests.
In relation to orders, we need to keep all relevant info (including IP) for 5 years.
Indeed. The advice from our lawyers is that you need to have a policy, say what it is, and follow it. The retention period we're going for is... don't laugh... 6 years. It's a US company that really couldn't give a sh*# about GDPR. I guess the server guys hold the logs for that long already so that's the number they want in our policy. Keeps it simple. It might get challenged when some EU customers read the policy.
EU data subjects can challenge you on your retention policy, if not happy with it report it to a supervisory authority in an EU country, and then they will investigate. If you cannot adequately justify it then you could end up being fined. I would suggest a more humble approach.
"So we have decided, with lawyers, that we have a lawfull purpose to keep the logs for 24 months."
...
"The retention period we're going for is... don't laugh... 6 years."
The retention period itself isn't the main factor. It's what you're doing with the logs in that time that really matters, and how you enforce that pattern of use.
Let's say we've got two companies. Company 1 decides to follow this draft and hold the logs for three days. But they do nothing to secure them and actively provide the information to their data analysis and marketing teams to be exploited to hell for that three days before deletion.
Company 2 decides to hold the logs for a probably ludicrous 10 years, but writes them to archival WORM storage, protected by several layers of technical and organisational process that is only used to specifically respond to suspected breaches.
Company 2 is almost certainly GDPR compliant. Company 1 is definitely not.
>The retention period itself isn't the main factor. It's what you're doing with the logs in that time that really >matters, and how you enforce that pattern of use.
No that is not right.
You can do whatever you want with the logs so long as the data subjects whose data in those logs have given you consent. One of the lawful reasons that you can provide for processing data is "Consent". The other main lawful reason is "To satisfy the performance under a contract", in other words, you are collecting, processing (which includes storing) the personal data into order to deliver the service to them.
What you can't do is, collect PII data from a user, tell them you are collecting it for it to be used for a particular purpose, and then later, do something different with the data which the user doesn't know about. If you want to do something new with the data, use it in a new process or for some other purpose, you need to go back to the user (data subject) and ask for their permission.
The retention time issue comes under a different principle of GDPR. And it is a fundamentally important principle of GDPR. You should only keep PII data for as long as is necessary, and you need to be able to justify why you are keeping it for that length of time.
Just curious. In the US the IRS can stick a probe in you for seven years, standard. Longer if they are pissed off. You better have receipts to back up every jot and tittle on your tax forms, and I suppose these would be covered if we had a GDPR-like law. Not only do I have proper names in them, but for some these are combined with websites, snail locations, etc. How does a European keep their personal or corporate financials on the right side of the law now?
For the record, when I dump a log, I want some privacy. Whether I get it... Who knows.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
