Extracting Every Email Address or URL from a Block of Text

Whose data this is

This article is written for people processing text they already own — a contact export from a system you administer, a list of links out of your own document, a set of addresses a client sent you and asked you to tidy. Collecting addresses from other people's web pages to build a mailing list is a different activity with its own legal exposure: under the General Data Protection Regulation an email address that identifies a person is personal data whatever page it was published on, and the anti-spam rules that govern unsolicited commercial mail apply on top of that. That is the whole of what will be said about it here.

The practical reason to say it at all is that both tools run entirely in the page, on your machine, with nothing uploaded. That is a real privacy property when the text is a client list, and it is worth knowing about. It is not a legal permission, and the two things get conflated often enough to be worth separating in one sentence before moving on to what the tools actually do.

Where a URL ends is a guess, and this one guesses on five characters

The WHATWG URL Standard defines what a URL is and how to parse one you have already isolated. It does not define how to find one inside a paragraph of prose, because that question has no correct answer: a full stop after a URL is punctuation to a reader and a legal path character to a parser, and nothing in the bytes distinguishes them. Every extractor therefore picks a set of characters at which to stop. This one stops at whitespace, a double quote, an apostrophe, either angle bracket, and a closing round bracket, and it was checked against fifteen contexts to see where that lands.

Excluding the closing round bracket buys two important cases. A link in parentheses comes out clean, and so does a Markdown link written as text in square brackets followed by the address in round ones. It costs one case, and the cost is worse than the benefit is good: an address that legitimately contains brackets in its path — the Wikipedia disambiguation convention is the everyday example — is cut short at the opening bracket's partner and returned truncated. A truncated URL still looks like a URL. It goes into a spreadsheet, it gets clicked, and it produces a page-not-found weeks later with nothing to trace it back to.

The characters it does not exclude are the ones that punctuate ordinary writing. A URL at the end of a sentence came back with the full stop attached. So did one followed by a comma, one followed by a semicolon, one followed by an exclamation mark, one inside square brackets, one inside typographic curly quotes, one wrapped in Markdown underscores and one in backticks. French guillemets happen to survive, but only because they are conventionally set with a space on each side, and the space is what stops the match. None of these produce an error; they produce a slightly wrong string that has to be cleaned by hand.

The upper-case link that is never found

The pattern is written in lower case and carries no case-insensitive flag. Text containing HTTPS://EXAMPLE.COM/A returned nothing at all. So did Https://example.com/a with a single capital, which is what some word processors produce when a link starts a sentence and autocapitalisation is on. This is not a boundary case; it is a whole class of document — press releases, legal notices, anything typed in a house style that shouts — from which this tool will extract zero links and report no problem. The URL Standard is explicit that the scheme is compared after lower-casing, so a scheme in capitals is a normal, valid URL, not a malformed one.

Three more absences are by design rather than by accident, and it helps to know which is which. A bare address such as www followed by a domain has no scheme, so it does not match — and cannot, without the tool guessing that any word with dots in it is a host. An ftp link does not match. A mailto link does not match, although extract-emails will happily pull the address out of it. If your source is a page of contact details written by a human rather than a machine, expect the bare addresses to be the majority and plan a second pass with extract-domain, which does look for them.

The accented address that comes back wrong

extract-emails looks for a run of letters, digits, dots, underscores, percent signs, plus signs and hyphens, then an at sign, then a host, then a dot and at least two letters. A plus tag survives, which matters because that is how many people label their subscriptions. A trailing full stop is correctly left out, because the pattern has to finish on letters — the same sentence that gives a URL an unwanted full stop gives an address a clean ending, which is a nice illustration of how much the shape of the pattern decides.

The local part is ASCII only, and that is where it turns from incomplete into wrong. Because a regular expression looks for a match anywhere in the line rather than requiring the whole token, an address whose first letter is accented does not fail to match — it matches from just after the accented letter. A German first name with an umlaut produced an address consisting of the last two letters of the name plus the at sign and the host. A French name with a cedilla lost its first three letters. A Portuguese name with a tilde lost everything but its last letter. And a surname with an apostrophe returned only the part after the apostrophe. Each of those is a syntactically valid address that belongs to somebody else, or to nobody, and nothing in the output marks it.

The domain half has the mirror-image problem and fails more safely. A host written with accented letters in its own alphabet — the internationalised domain names that RFC 5890 defines and RFC 6532 allows in mail headers — does not match at all, so the address is skipped rather than mangled. Written in its punycode form, beginning with the x-n prefix, the same host matches fine, because punycode is ASCII by construction. Two smaller notes from the runs: a quoted local part, which RFC 5322 permits, is not recognised; and a mixed-case address and its lower-case twin both survive the de-duplication, because the comparison is exact and the domain half of an address is case-insensitive by specification, so those two are one mailbox listed twice.

Domains and phone numbers, briefly

extract-domain covers the gap left by the URL extractor: it reads hosts out of schemed links, out of email addresses, and — with a toggle that is on by default — out of bare domains written with no scheme at all. It strips the user and password in front of a host, the port behind it, the trailing root dot and, optionally, a leading www. All of that was confirmed on a line containing a login-carrying intranet address. It also refuses a list of file extensions so that an image or a page file does not read as a domain, and the ones on the list do get skipped.

Two defects worth reporting. First, a bare domain at the end of a sentence is missed entirely: the pattern that finds bare domains refuses to stop in front of a full stop, so a sentence ending in a domain followed by a period returned nothing, while the same domain in the middle of a sentence, or before a comma, or before an exclamation mark, was found. That is the commonest way a domain appears in prose, and it is the one case it does not handle. Second, the file-extension list is a fixed list rather than a rule, so extensions that are not on it are read as domains: an archive file and a video file both came back as though they were hosts. Neither problem produces an error message.

extract-phone-numbers is the most cautious of the four. It matches four specific shapes rather than any run of digits, requires between seven and fifteen digits after the punctuation is stripped, and was checked to leave dates alone in both the year-month-day and the day-slash-month forms, and to leave a long reference number alone. What it cannot do is tell a phone number from something that is shaped exactly like one — an order reference of three digits, three digits and four digits came back as a North American number, and there is no signal in the text to say otherwise. Read its output rather than trusting it, which is the sentence this whole article keeps arriving at.

Frequently asked questions

Why did my extracted links come out with a full stop or a comma on the end?

Because the extractor stops at whitespace, quotes, angle brackets and a closing round bracket, and at nothing else. Sentence punctuation is a legal path character as far as the pattern is concerned, so the full stop that ends your sentence becomes the last character of the link. There is no setting for it. The practical fix is a find-and-replace pass over the extracted list, or a manual trim if the list is short. The same problem does not affect extracted email addresses, because their pattern has to finish on letters, so a trailing full stop cannot be part of a match.

Some links in my document were not found at all. What did they have in common?

Check the scheme first. A link whose scheme carries any capital letter is invisible to this tool — an upper-case address returned nothing at all when tested, and so did one with a single leading capital, which is what a word processor produces when a link opens a sentence. Then check for links with no scheme: a bare address starting with www is not matched, because the pattern requires the protocol. And ftp and mailto links are outside its remit by design. Lower-casing the document before extracting fixes the first problem; extract-domain, which does look for bare hosts, covers most of the second.

Are the addresses it returns safe to use as they are?

Not without a look. The extractor guarantees that each result matches its pattern, not that each result is an address a person owns. Two failure modes were observed directly: an address whose local part begins with an accented letter is returned truncated, which produces a valid-looking address that is not the one in the text; and the credentials in a URL of the form user, colon, password, at sign, host are returned as though they were an address. On a short list, read it. On a long one, sort it and look at the ends — truncated addresses cluster at the top because they are short, and the fabricated ones tend to share a host you recognise as a server rather than a mail provider.

Does it decode the ampersand entity in a query string?

No. A URL copied out of HTML source, where the separator between two parameters is written as the ampersand entity rather than as the character, comes back with the entity intact — it was tested and the five characters are preserved exactly. The result is a link that will be requested with parameter names beginning with amp semicolon, which usually means the second parameter is silently ignored by the server. If your source is HTML rather than rendered text, decode the entities first; if it is text you copied from a rendered page, the browser has already decoded them and there is nothing to do.

Can I use these tools on addresses I collected from other people's websites?

The tools will process whatever you paste; whether you may collect and use those addresses is a separate question and not a technical one. In the European Union an email address that identifies a person is personal data under the General Data Protection Regulation regardless of the page it appeared on, and the rules on unsolicited commercial mail apply independently of that. In the United States the equivalent commercial-mail rules are federal. If the list is yours — your own export, your own document, a client's data you were engaged to clean — none of this is in question, and that is the situation this article was written for. If it is not, the question to answer is a legal one, and it is not answered by the fact that the page was public.