Fetch
JavaScript can send network requests to the server and load new information whenever it's needed. For example, we can use a network request to: - Submit an order, - Load user information, - Receive latest updates from the server, - ...etc. ...And all of that without reloading the page! There's an umbrella term "AJAX" (abbreviated <b>A</b>synchronous <b>J</b>avaScript <b>A</b>nd <b>X</b>ML) for network requests from JavaScript. We don't have to use XML though: the term comes from old times, that's why that word is there. You may have heard that term already. There are multiple ways to send a network request and get information from the server. The fetch() method is modern and versatile, so we'll start with it. It's not supported by old browsers (can be polyfilled), but very well supported among the modern ones. The basic syntax is: - url -- the URL to access. - options -- optional parameters: method, headers etc. Without options, this is a simple GET request, downloading the contents of the url. The browser starts the request right away and returns a promise that the calling code should use to get the result. Getting a response is usually a two-stage process. First, the promise, returned by fetch, resolves with an object of the built-in Response class as soon as the server responds with headers. At this stage we can check HTTP status, to see whether it is successful or not, check headers, but don't have the body yet. The promise rejects if the fetch was unable to make HTTP-request, e.g. network problems, or there's no such site. Abnormal HTTP-statuses, such as 404 or 500 do not cause an error. We can see HTTP-status in response properties: - status -- HTTP status code, e.g. 200. - ok -- boolean, true if the HTTP status code is 200-299. For example: Second, to get the response body, we need to use an additional method call. Response provides multiple promise-based methods to access the body in various formats: - response.text() -- read the response and return as text, - response.json() -- parse the response as JSON, - response.formData() -- return the response as FormData object (explained in the next chapter), - response.blob() -- return the response as Blob (binary data with type), - response.arrayBuffer() -- return the response as ArrayBuffer (low-level representation of binary data), - additionally, response.body is a ReadableStream object, it allows you to read the body chunk-by-chunk, we'll see an example later. For instance, let's get a JSON-object with latest commits from GitHub: Or, the same without await, using pure promises syntax: To get the response text, await response.text() instead of .json(): As a show-case for reading in binary format, let's fetch and show a logo image of "fetch" specification (see chapter Blob for details about operations on Blob): let text = await response.text(); // response body consumed let parsed = await response.json(); // fails (already consumed) ## Response headers The response headers are available in a Map-like headers object in response.headers. It's not exactly a Map, but it has similar methods to get individual headers by name or iterate over them: ## Request headers To set a request header in fetch, we can use the headers option. It has an object with outgoing headers, like this: ...But there's a list of forbidden HTTP headers that we can't set: - Accept-Charset, Accept-Encoding - Access-Control-Request-Headers - Access-Control-Request-Method - Connection - Content-Length - Cookie, Cookie2 - Date - DNT - Expect - Host - Keep-Alive - Origin - Referer - TE - Trailer - Transfer-Encoding - Upgrade - Via - Proxy-* - Sec-* These headers ensure proper and safe HTTP, so they are controlled exclusively by the browser. ## POST requests To make a POST request, or a request with another method, we need to use fetch options: - method -- HTTP-method, e.g. POST, - body -- the request body, one of: - a string (e.g. JSON-encoded), - FormData object, to submit the data as multipart/form-data, - Blob/BufferSource to send binary data, - URLSearchParams, to submit the data in x-www-form-urlencoded encoding, rarely used. The JSON format is used most of the time. For example, this code submits user object as JSON: Please note, if the request body is a string, then Content-Type header is set to text/plain;charset=UTF-8 by default. But, as we're going to send JSON, we use headers option to send application/json instead, the correct Content-Type for JSON-encoded data. ## Sending an image We can also submit binary data with fetch using Blob or BufferSource objects. In this example, there's a <canvas> where we can draw by moving a mouse over it. A click on the "submit" button sends the image to the server: Please note, here we don't set Content-Type header manually, because a Blob object has a built-in type (here image/png, as generated by toBlob). For Blob objects that type becomes the value of Content-Type. The submit() function can be rewritten without async/await like this: ## Summary A typical fetch request consists of two await calls: Or, without await: Response properties: - response.status -- HTTP code of the response, - response.ok -- true if the status is 200-299. - response.headers -- Map-like object with HTTP headers. Methods to get response body: - response.text() -- return the response as text, - response.json() -- parse the response as JSON object, - response.formData() -- return the response as FormData object (multipart/form-data encoding, see the next chapter), - response.blob() -- return the response as Blob (binary data with type), - response.arrayBuffer() -- return the response as ArrayBuffer (low-level binary data), Fetch options so far: - method -- HTTP-method, - headers -- an object with request headers (not any header is allowed), - body -- the data to send (request body) as string, FormData, BufferSource, Blob or UrlSearchParams object. In the next chapters we'll see more options and use cases of fetch.
FormData
This chapter is about sending HTML forms: with or without files, with additional fields and so on. FormData objects can help with that. As you might have guessed, it's the object to represent HTML form data. The constructor is: If HTML form element is provided, it automatically captures its fields. The special thing about FormData is that network methods, such as fetch, can accept a FormData object as a body. It's encoded and sent out with Content-Type: multipart/form-data. From the server point of view, that looks like a usual form submission. ## Sending a simple form Let's send a simple form first. As you can see, that's almost one-liner: In this example, the server code is not presented, as it's beyond our scope. The server accepts the POST request and replies "User saved". ## FormData Methods We can modify fields in FormData with methods: - formData.append(name, value) - add a form field with the given name and value, - formData.append(name, blob, fileName) - add a field as if it were <input type="file">, the third argument fileName sets file name (not form field name), as it were a name of the file in user's filesystem, - formData.delete(name) - remove the field with the given name, - formData.get(name) - get the value of the field with the given name, - formData.has(name) - if there exists a field with the given name, returns true, otherwise false A form is technically allowed to have many fields with the same name, so multiple calls to append add more same-named fields. There's also method set, with the same syntax as append. The difference is that .set removes all fields with the given name, and then appends a new field. So it makes sure there's only one field with such name, the rest is just like append: - formData.set(name, value), - formData.set(name, blob, fileName). Also we can iterate over formData fields using for..of loop: ## Sending a form with a file The form is always sent as Content-Type: multipart/form-data, this encoding allows to send files. So, <input type="file"> fields are sent also, similar to a usual form submission. Here's an example with such form: ## Sending a form with Blob data As we've seen in the chapter <info:fetch>, it's easy to send dynamically generated binary data e.g. an image, as Blob. We can supply it directly as fetch parameter body. In practice though, it's often convenient to send an image not separately, but as a part of the form, with additional fields, such as "name" and other metadata. Also, servers are usually more suited to accept multipart-encoded forms, rather than raw binary data. This example submits an image from <canvas>, along with some other fields, as a form, using FormData: Please note how the image Blob is added: That's same as if there were <input type="file" name="image"> in the form, and the visitor submitted a file named "image.png" (3rd argument) with the data imageBlob (2nd argument) from their filesystem. The server reads form data and the file, as if it were a regular form submission. ## Summary FormData objects are used to capture HTML form and submit it using fetch or another network method. We can either create new FormData(form) from an HTML form, or create an object without a form at all, and then append fields with methods: - formData.append(name, value) - formData.append(name, blob, fileName) - formData.set(name, value) - formData.set(name, blob, fileName) Let's note two peculiarities here: 1. The set method removes fields with the same name, append doesn't. That's the only difference between them. 2. To send a file, 3-argument syntax is needed, the last argument is a file name, that normally is taken from user filesystem for <input type="file">. Other methods are: - formData.delete(name) - formData.get(name) - formData.has(name) That's it!
Fetch: Download progress
The fetch method allows to track download progress. Please note: there's currently no way for fetch to track upload progress. For that purpose, please use XMLHttpRequest, we'll cover it later. To track download progress, we can use response.body property. It's a ReadableStream -- a special object that provides body chunk-by-chunk, as it comes. Readable streams are described in the Streams API specification. Unlike response.text(), response.json() and other methods, response.body gives full control over the reading process, and we can count how much is consumed at any moment. Here's the sketch of code that reads the response from response.body: The result of await reader.read() call is an object with two properties: - done -- true when the reading is complete, otherwise false. - value -- a typed array of bytes: Uint8Array. We receive response chunks in the loop, until the loading finishes, that is: until done becomes true. To log the progress, we just need for every received fragment value to add its length to the counter. Here's the full working example that gets the response and logs the progress in console, more explanations to follow: Let's explain that step-by-step: 1. We perform fetch as usual, but instead of calling response.json(), we obtain a stream reader response.body.getReader(). Please note, we can't use both these methods to read the same response: either use a reader or a response method to get the result. 2. Prior to reading, we can figure out the full response length from the Content-Length header. It may be absent for cross-origin requests (see chapter <info:fetch-crossorigin>) and, well, technically a server doesn't have to set it. But usually it's at place. 3. Call await reader.read() until it's done. We gather response chunks in the array chunks. That's important, because after the response is consumed, we won't be able to "re-read" it using response.json() or another way (you can try, there'll be an error). 4. At the end, we have chunks -- an array of Uint8Array byte chunks. We need to join them into a single result. Unfortunately, there's no single method that concatenates those, so there's some code to do that: 1. We create chunksAll = new Uint8Array(receivedLength) -- a same-typed array with the combined length. 2. Then use .set(chunk, position) method to copy each chunk one after another in it. 5. We have the result in chunksAll. It's a byte array though, not a string. To create a string, we need to interpret these bytes. The built-in TextDecoder does exactly that. Then we can JSON.parse it, if necessary. What if we need binary content instead of a string? That's even simpler. Replace steps 4 and 5 with a single line that creates a Blob from all chunks: ```js let blob = new Blob(chunks); ``` At the end we have the result (as a string or a blob, whatever is convenient), and progress-tracking in the process. Once again, please note, that's not for upload progress (no way now with fetch), only for download progress. Also, if the size is unknown, we should check receivedLength in the loop and break it once it reaches a certain limit. So that the chunks won't overflow the memory.
Fetch: Abort
As we know, fetch returns a promise. And JavaScript generally has no concept of "aborting" a promise. So how can we cancel an ongoing fetch? E.g. if the user actions on our site indicate that the fetch isn't needed any more. There's a special built-in object for such purposes: AbortController. It can be used to abort not only fetch, but other asynchronous tasks as well. The usage is very straightforward: ## The AbortController object Create a controller: A controller is an extremely simple object. - It has a single method abort(), - And a single property signal that allows to set event listeners on it. When abort() is called: - controller.signal emits the "abort" event. - controller.signal.aborted property becomes true. Generally, we have two parties in the process: 1. The one that performs a cancelable operation, it sets a listener on controller.signal. 2. The one that cancels: it calls controller.abort() when needed. Here's the full example (without fetch yet): As we can see, AbortController is just a mean to pass abort events when abort() is called on it. We could implement the same kind of event listening in our code on our own, without the AbortController object. But what's valuable is that fetch knows how to work with the AbortController object. It's integrated in it. ## Using with fetch To be able to cancel fetch, pass the signal property of an AbortController as a fetch option: The fetch method knows how to work with AbortController. It will listen to abort events on signal. Now, to abort, call controller.abort(): We're done: fetch gets the event from signal and aborts the request. When a fetch is aborted, its promise rejects with an error AbortError, so we should handle it, e.g. in try..catch. Here's the full example with fetch aborted after 1 second: ## AbortController is scalable AbortController is scalable. It allows to cancel multiple fetches at once. Here's a sketch of code that fetches many urls in parallel, and uses a single controller to abort them all: If we have our own asynchronous tasks, different from fetch, we can use a single AbortController to stop those, together with fetches. We just need to listen to its abort event in our tasks: ## Summary - AbortController is a simple object that generates an abort event on its signal property when the abort() method is called (and also sets signal.aborted to true). - fetch integrates with it: we pass the signal property as the option, and then fetch listens to it, so it's possible to abort the fetch. - We can use AbortController in our code. The "call abort()" -> "listen to abort event" interaction is simple and universal. We can use it even without fetch.
Fetch: Cross-Origin Requests
If we send a fetch request to another web-site, it will probably fail. For instance, let's try fetching http://example.com: Fetch fails, as expected. The core concept here is origin -- a domain/port/protocol triplet. Cross-origin requests -- those sent to another domain (even a subdomain) or protocol or port -- require special headers from the remote side. That policy is called "CORS": Cross-Origin Resource Sharing. ## Why is CORS needed? A brief history CORS exists to protect the internet from evil hackers. Seriously. Let's make a very brief historical digression. For many years a script from one site could not access the content of another site. That simple, yet powerful rule was a foundation of the internet security. E.g. an evil script from website hacker.com could not access the user's mailbox at website gmail.com. People felt safe. JavaScript also did not have any special methods to perform network requests at that time. It was a toy language to decorate a web page. But web developers demanded more power. A variety of tricks were invented to work around the limitation and make requests to other websites. ### Using forms One way to communicate with another server was to submit a <form> there. People submitted it into <iframe>, just to stay on the current page, like this: So, it was possible to make a GET/POST request to another site, even without networking methods, as forms can send data anywhere. But as it's forbidden to access the content of an <iframe> from another site, it wasn't possible to read the response. To be precise, there were actually tricks for that, they required special scripts at both the iframe and the page. So the communication with the iframe was technically possible. Right now there's no point to go into details, let these dinosaurs rest in peace. ### Using scripts Another trick was to use a script tag. A script could have any src, with any domain, like <script src="http://another.com/…">. It's possible to execute a script from any website. If a website, e.g. another.com intended to expose data for this kind of access, then a so-called "JSONP (JSON with padding)" protocol was used. Here's how it worked. Let's say we, at our site, need to get the data from http://another.com, such as the weather: 1. First, in advance, we declare a global function to accept the data, e.g. gotWeather. ```js // 1. Declare the function to process the weather data function gotWeather({ temperature, humidity }) { alert(temperature: ${temperature}, humidity: ${humidity}); ``` 2. Then we make a <script> tag with src="http://another.com/weather.json?callback=gotWeather", using the name of our function as the callback URL-parameter. ```js let script = document.createElement('script'); script.src = http://another.com/weather.json?callback=gotWeather; document.body.append(script); ``` 3. The remote server another.com dynamically generates a script that calls gotWeather(...) with the data it wants us to receive. ```js // The expected answer from the server looks like this: gotWeather({ temperature: 25, humidity: 78 }); ``` 4. When the remote script loads and executes, gotWeather runs, and, as it's our function, we have the data. That works, and doesn't violate security, because both sides agreed to pass the data this way. And, when both sides agree, it's definitely not a hack. There are still services that provide such access, as it works even for very old browsers. After a while, networking methods appeared in browser JavaScript. At first, cross-origin requests were forbidden. But as a result of long discussions, cross-origin requests were allowed, but with any new capabilities requiring an explicit allowance by the server, expressed in special headers. ## Safe requests There are two types of cross-origin requests: 1. Safe requests. 2. All the others. Safe Requests are simpler to make, so let's start with them. A request is safe if it satisfies two conditions: 1. Safe method: GET, POST or HEAD 2. Safe headers -- the only allowed custom headers are: - Accept, - Accept-Language, - Content-Language, - Content-Type with the value application/x-www-form-urlencoded, multipart/form-data or text/plain. Any other request is considered "unsafe". For instance, a request with PUT method or with an API-Key HTTP-header does not fit the limitations. The essential difference is that a safe request can be made with a <form> or a <script>, without any special methods. So, even a very old server should be ready to accept a safe request. Contrary to that, requests with non-standard headers or e.g. method DELETE can't be created this way. For a long time JavaScript was unable to do such requests. So an old server may assume that such requests come from a privileged source, "because a webpage is unable to send them". When we try to make a unsafe request, the browser sends a special "preflight" request that asks the server -- does it agree to accept such cross-origin requests, or not? And, unless the server explicitly confirms that with headers, an unsafe request is not sent. Now we'll go into details. ## CORS for safe requests If a request is cross-origin, the browser always adds the Origin header to it. For instance, if we request https://anywhere.com/request from https://javascript.info/page, the headers will look like: As you can see, the Origin header contains exactly the origin (domain/protocol/port), without a path. The server can inspect the Origin and, if it agrees to accept such a request, add a special header Access-Control-Allow-Origin to the response. That header should contain the allowed origin (in our case https://javascript.info), or a star *. Then the response is successful, otherwise it's an error. The browser plays the role of a trusted mediator here: 1. It ensures that the correct Origin is sent with a cross-origin request. 2. It checks for permitting Access-Control-Allow-Origin in the response, if it exists, then JavaScript is allowed to access the response, otherwise it fails with an error. Here's an example of a permissive server response: ## Response headers For cross-origin request, by default JavaScript may only access so-called "safe" response headers: - Cache-Control - Content-Language - Content-Length - Content-Type - Expires - Last-Modified - Pragma Accessing any other response header causes an error. To grant JavaScript access to any other response header, the server must send the Access-Control-Expose-Headers header. It contains a comma-separated list of unsafe header names that should be made accessible. For example: With such an Access-Control-Expose-Headers header, the script is allowed to read the Content-Encoding and API-Key headers of the response. ## "Unsafe" requests We can use any HTTP-method: not just GET/POST, but also PATCH, DELETE and others. Some time ago no one could even imagine that a webpage could make such requests. So there may still exist webservices that treat a non-standard method as a signal: "That's not a browser". They can take it into account when checking access rights. So, to avoid misunderstandings, any "unsafe" request -- that couldn't be done in the old times, the browser does not make such requests right away. First, it sends a preliminary, so-called "preflight" request, to ask for permission. A preflight request uses the method OPTIONS, no body and three headers: - Access-Control-Request-Method header has the method of the unsafe request. - Access-Control-Request-Headers header provides a comma-separated list of its unsafe HTTP-headers. - Origin header tells from where the request came. (such as https://javascript.info) If the server agrees to serve the requests, then it should respond with empty body, status 200 and headers: - Access-Control-Allow-Origin must be either * or the requesting origin, such as https://javascript.info, to allow it. - Access-Control-Allow-Methods must have the allowed method. - Access-Control-Allow-Headers must have a list of allowed headers. - Additionally, the header Access-Control-Max-Age may specify a number of seconds to cache the permissions. So the browser won't have to send a preflight for subsequent requests that satisfy given permissions. Let's see how it works step-by-step on the example of a cross-origin PATCH request (this method is often used to update data): There are three reasons why the request is unsafe (one is enough): - Method PATCH - Content-Type is not one of: application/x-www-form-urlencoded, multipart/form-data, text/plain. - "Unsafe" API-Key header. ### Step 1 (preflight request) Prior to sending such a request, the browser, on its own, sends a preflight request that looks like this: - Method: OPTIONS. - The path -- exactly the same as the main request: /service.json. - Cross-origin special headers: - Origin -- the source origin. - Access-Control-Request-Method -- requested method. - Access-Control-Request-Headers -- a comma-separated list of "unsafe" headers. ### Step 2 (preflight response) The server should respond with status 200 and the headers: - Access-Control-Allow-Origin: https://javascript.info - Access-Control-Allow-Methods: PATCH - Access-Control-Allow-Headers: Content-Type,API-Key. That allows future communication, otherwise an error is triggered. If the server expects other methods and headers in the future, it makes sense to allow them in advance by adding them to the list. For example, this response also allows PUT, DELETE and additional headers: Now the browser can see that PATCH is in Access-Control-Allow-Methods and Content-Type,API-Key are in the list Access-Control-Allow-Headers, so it sends out the main request. If there's the header Access-Control-Max-Age with a number of seconds, then the preflight permissions are cached for the given time. The response above will be cached for 86400 seconds (one day). Within this timeframe, subsequent requests will not cause a preflight. Assuming that they fit the cached allowances, they will be sent directly. ### Step 3 (actual request) When the preflight is successful, the browser now makes the main request. The process here is the same as for safe requests. The main request has the Origin header (because it's cross-origin): ### Step 4 (actual response) The server should not forget to add Access-Control-Allow-Origin to the main response. A successful preflight does not relieve from that: Then JavaScript is able to read the main server response. ## Credentials A cross-origin request initiated by JavaScript code by default does not bring any credentials (cookies or HTTP authentication). That's uncommon for HTTP-requests. Usually, a request to http://site.com is accompanied by all cookies from that domain. Cross-origin requests made by JavaScript methods on the other hand are an exception. For example, fetch('http://another.com') does not send any cookies, even those (!) that belong to another.com domain. Why? That's because a request with credentials is much more powerful than without them. If allowed, it grants JavaScript the full power to act on behalf of the user and access sensitive information using their credentials. Does the server really trust the script that much? Then it must explicitly allow requests with credentials with an additional header. To send credentials in fetch, we need to add the option credentials: "include", like this: Now fetch sends cookies originating from another.com with request to that site. If the server agrees to accept the request with credentials, it should add a header Access-Control-Allow-Credentials: true to the response, in addition to Access-Control-Allow-Origin. For example: Please note: Access-Control-Allow-Origin is prohibited from using a star * for requests with credentials. Like shown above, it must provide the exact origin there. That's an additional safety measure, to ensure that the server really knows who it trusts to make such requests. ## Summary From the browser point of view, there are two kinds of cross-origin requests: "safe" and all the others. "Safe" requests must satisfy the following conditions: - Method: GET, POST or HEAD. - Headers -- we can set only: - Accept - Accept-Language - Content-Language - Content-Type to the value application/x-www-form-urlencoded, multipart/form-data or text/plain. The essential difference is that safe requests were doable since ancient times using <form> or <script> tags, while unsafe were impossible for browsers for a long time. So, the practical difference is that safe requests are sent right away, with the Origin header, while for the other ones the browser makes a preliminary "preflight" request, asking for permission. For safe requests: - → The browser sends the Origin header with the origin. - ← For requests without credentials (not sent by default), the server should set: - Access-Control-Allow-Origin to * or same value as Origin - ← For requests with credentials, the server should set: - Access-Control-Allow-Origin to same value as Origin - Access-Control-Allow-Credentials to true Additionally, to grant JavaScript access to any response headers except Cache-Control, Content-Language, Content-Type, Expires, Last-Modified or Pragma, the server should list the allowed ones in Access-Control-Expose-Headers header. For unsafe requests, a preliminary "preflight" request is issued before the requested one: - → The browser sends an OPTIONS request to the same URL, with the headers: - Access-Control-Request-Method has requested method. - Access-Control-Request-Headers lists unsafe requested headers. - ← The server should respond with status 200 and the headers: - Access-Control-Allow-Methods with a list of allowed methods, - Access-Control-Allow-Headers with a list of allowed headers, - Access-Control-Max-Age with a number of seconds to cache the permissions. - Then the actual request is sent, and the previous "safe" scheme is applied.
Fetch API
So far, we know quite a bit about fetch. Let's see the rest of API, to cover all its abilities. Here's the full list of all possible fetch options with their default values (alternatives in comments): An impressive list, right? We fully covered method, headers and body in the chapter <info:fetch>. The signal option is covered in <info:fetch-abort>. Now let's explore the remaining capabilities. ## referrer, referrerPolicy These options govern how fetch sets the HTTP Referer header. Usually that header is set automatically and contains the url of the page that made the request. In most scenarios, it's not important at all, sometimes, for security purposes, it makes sense to remove or shorten it. The referrer option allows to set any Referer (within the current origin) or remove it. To send no referrer, set an empty string: To set another url within the current origin: The referrerPolicy option sets general rules for Referer. Requests are split into 3 types: 1. Request to the same origin. 2. Request to another origin. 3. Request from HTTPS to HTTP (from safe to unsafe protocol). Unlike the referrer option that allows to set the exact Referer value, referrerPolicy tells the browser general rules for each request type. Possible values are described in the Referrer Policy specification: - "strict-origin-when-cross-origin" -- the default value: for same-origin send the full Referer, for cross-origin send only the origin, unless it's HTTPS→HTTP request, then send nothing. - "no-referrer-when-downgrade" -- full Referer is always sent, unless we send a request from HTTPS to HTTP (to the less secure protocol). - "no-referrer" -- never send Referer. - "origin" -- only send the origin in Referer, not the full page URL, e.g. only http://site.com instead of http://site.com/path. - "origin-when-cross-origin" -- send the full Referer to the same origin, but only the origin part for cross-origin requests (as above). - "same-origin" -- send the full Referer to the same origin, but no Referer for cross-origin requests. - "strict-origin" -- send only the origin, not the Referer for HTTPS→HTTP requests. - "unsafe-url" -- always send the full url in Referer, even for HTTPS→HTTP requests. Here's a table with all combinations: Let's say we have an admin zone with a URL structure that shouldn't be known from outside of the site. If we send a fetch, then by default it always sends the Referer header with the full url of our page (except when we request from HTTPS to HTTP, then no Referer). E.g. Referer: https://javascript.info/admin/secret/paths. If we'd like other websites know only the origin part, not the URL-path, we can set the option: We can put it to all fetch calls, maybe integrate into JavaScript library of our project that does all requests and uses fetch inside. Its only difference compared to the default behavior is that for requests to another origin fetch sends only the origin part of the URL (e.g. https://javascript.info, without path). For requests to our origin we still get the full Referer (maybe useful for debugging purposes). ## mode The mode option is a safe-guard that prevents occasional cross-origin requests: - "cors" -- the default, cross-origin requests are allowed, as described in <info:fetch-crossorigin>, - "same-origin" -- cross-origin requests are forbidden, - "no-cors" -- only safe cross-origin requests are allowed. This option may be useful when the URL for fetch comes from a 3rd-party, and we want a "power off switch" to limit cross-origin capabilities. ## credentials The credentials option specifies whether fetch should send cookies and HTTP-Authorization headers with the request. - "same-origin" -- the default, don't send for cross-origin requests, - "include" -- always send, requires Access-Control-Allow-Credentials from cross-origin server in order for JavaScript to access the response, that was covered in the chapter <info:fetch-crossorigin>, - "omit" -- never send, even for same-origin requests. ## cache By default, fetch requests make use of standard HTTP-caching. That is, it respects the Expires and Cache-Control headers, sends If-Modified-Since and so on. Just like regular HTTP-requests do. The cache options allows to ignore HTTP-cache or fine-tune its usage: - "default" -- fetch uses standard HTTP-cache rules and headers, - "no-store" -- totally ignore HTTP-cache, this mode becomes the default if we set a header If-Modified-Since, If-None-Match, If-Unmodified-Since, If-Match, or If-Range, - "reload" -- don't take the result from HTTP-cache (if any), but populate the cache with the response (if the response headers permit this action), - "no-cache" -- create a conditional request if there is a cached response, and a normal request otherwise. Populate HTTP-cache with the response, - "force-cache" -- use a response from HTTP-cache, even if it's stale. If there's no response in HTTP-cache, make a regular HTTP-request, behave normally, - "only-if-cached" -- use a response from HTTP-cache, even if it's stale. If there's no response in HTTP-cache, then error. Only works when mode is "same-origin". ## redirect Normally, fetch transparently follows HTTP-redirects, like 301, 302 etc. The redirect option allows to change that: - "follow" -- the default, follow HTTP-redirects, - "error" -- error in case of HTTP-redirect, - "manual" -- allows to process HTTP-redirects manually. In case of redirect, we'll get a special response object, with response.type="opaqueredirect" and zeroed/empty status and most other properies. ## integrity The integrity option allows to check if the response matches the known-ahead checksum. As described in the specification, supported hash-functions are SHA-256, SHA-384, and SHA-512, there might be others depending on the browser. For example, we're downloading a file, and we know that its SHA-256 checksum is "abcdef" (a real checksum is longer, of course). We can put it in the integrity option, like this: Then fetch will calculate SHA-256 on its own and compare it with our string. In case of a mismatch, an error is triggered. ## keepalive The keepalive option indicates that the request may "outlive" the webpage that initiated it. For example, we gather statistics on how the current visitor uses our page (mouse clicks, page fragments he views), to analyze and improve the user experience. When the visitor leaves our page -- we'd like to save the data to our server. We can use the window.onunload event for that: Normally, when a document is unloaded, all associated network requests are aborted. But the keepalive option tells the browser to perform the request in the background, even after it leaves the page. So this option is essential for our request to succeed. It has a few limitations: - We can't send megabytes: the body limit for keepalive requests is 64KB. - If we need to gather a lot of statistics about the visit, we should send it out regularly in packets, so that there won't be a lot left for the last onunload request. - This limit applies to all keepalive requests together. In other words, we can perform multiple keepalive requests in parallel, but the sum of their body lengths should not exceed 64KB. - We can't handle the server response if the document is unloaded. So in our example fetch will succeed due to keepalive, but subsequent functions won't work. - In most cases, such as sending out statistics, it's not a problem, as the server just accepts the data and usually sends an empty response to such requests.
URL objects
The built-in URL class provides a convenient interface for creating and parsing URLs. There are no networking methods that require exactly a URL object, strings are good enough. So technically we don't have to use URL. But sometimes it can be really helpful. ## Creating a URL The syntax to create a new URL object: - url -- the full URL or only path (if base is set, see below), - base -- an optional base URL: if set and url argument has only path, then the URL is generated relative to base. For example: These two URLs are same: We can easily create a new URL based on the path relative to an existing URL: The URL object immediately allows us to access its components, so it's a nice way to parse the url, e.g.: Here's the cheatsheet for URL components: - href is the full url, same as url.toString() - protocol ends with the colon character : - search - a string of parameters, starts with the question mark ? - hash starts with the hash character # - there may be also user and password properties if HTTP authentication is present: http://login:[email protected] (not painted above, rarely used). ## SearchParams "?..." Let's say we want to create a url with given search params, for instance, https://google.com/search?query=JavaScript. We can provide them in the URL string: ...But parameters need to be encoded if they contain spaces, non-latin letters, etc (more about that below). So there's a URL property for that: url.searchParams, an object of type URLSearchParams. It provides convenient methods for search parameters: - append(name, value) -- add the parameter by name, - delete(name) -- remove the parameter by name, - get(name) -- get the parameter by name, - getAll(name) -- get all parameters with the same name (that's possible, e.g. ?user=John&user=Pete), - has(name) -- check for the existence of the parameter by name, - set(name, value) -- set/replace the parameter, - sort() -- sort parameters by name, rarely needed, - ...and it's also iterable, similar to Map. An example with parameters that contain spaces and punctuation marks: ## Encoding There's a standard RFC3986 that defines which characters are allowed in URLs and which are not. Those that are not allowed, must be encoded, for instance non-latin letters and spaces - replaced with their UTF-8 codes, prefixed by %, such as %20 (a space can be encoded by +, for historical reasons, but that's an exception). The good news is that URL objects handle all that automatically. We just supply all parameters unencoded, and then convert the URL to string: As you can see, both Тест in the url path and ъ in the parameter are encoded. The URL became longer, because each cyrillic letter is represented with two bytes in UTF-8, so there are two %.. entities. ### Encoding strings In old times, before URL objects appeared, people used strings for URLs. As of now, URL objects are often more convenient, but strings can still be used as well. In many cases using a string makes the code shorter. If we use a string though, we need to encode/decode special characters manually. There are built-in functions for that: - encodeURI - encodes URL as a whole. - decodeURI - decodes it back. - encodeURIComponent - encodes a URL component, such as a search parameter, or a hash, or a pathname. - decodeURIComponent - decodes it back. A natural question is: "What's the difference between encodeURIComponent and encodeURI? When we should use either?" That's easy to understand if we look at the URL, that's split into components in the picture above: As we can see, characters such as :, ?, =, &, # are allowed in URL. ...On the other hand, if we look at a single URL component, such as a search parameter, these characters must be encoded, not to break the formatting. - encodeURI encodes only characters that are totally forbidden in URL. - encodeURIComponent encodes same characters, and, in addition to them, characters #, $, &, +, ,, /, :, ;, =, ? and @. So, for a whole URL we can use encodeURI: ...While for URL parameters we should use encodeURIComponent instead: Compare it with encodeURI: As we can see, encodeURI does not encode &, as this is a legit character in URL as a whole. But we should encode & inside a search parameter, otherwise, we get q=Rock&Roll - that is actually q=Rock plus some obscure parameter Roll. Not as intended. So we should use only encodeURIComponent for each search parameter, to correctly insert it in the URL string. The safest is to encode both name and value, unless we're absolutely sure that it has only allowed characters. // valid url with IPv6 address let url = 'http://[2607:f8b0:4005:802::1007]/'; alert(encodeURI(url)); // http://%5B2607:f8b0:4005:802::1007%5D/ alert(new URL(url)); // http://[2607:f8b0:4005:802::1007]/
XMLHttpRequest
XMLHttpRequest is a built-in browser object that allows to make HTTP requests in JavaScript. Despite having the word "XML" in its name, it can operate on any data, not only in XML format. We can upload/download files, track progress and much more. Right now, there's another, more modern method fetch, that somewhat deprecates XMLHttpRequest. In modern web-development XMLHttpRequest is used for three reasons: 1. Historical reasons: we need to support existing scripts with XMLHttpRequest. 2. We need to support old browsers, and don't want polyfills (e.g. to keep scripts tiny). 3. We need something that fetch can't do yet, e.g. to track upload progress. Does that sound familiar? If yes, then all right, go on with XMLHttpRequest. Otherwise, please head on to <info:fetch>. ## The basics XMLHttpRequest has two modes of operation: synchronous and asynchronous. Let's see the asynchronous first, as it's used in the majority of cases. To do the request, we need 3 steps: 1. Create XMLHttpRequest: ```js let xhr = new XMLHttpRequest(); ``` The constructor has no arguments. 2. Initialize it, usually right after new XMLHttpRequest: ```js xhr.open(method, URL, [async, user, password]) ``` This method specifies the main parameters of the request: - method -- HTTP-method. Usually "GET" or "POST". - URL -- the URL to request, a string, can be URL object. - async -- if explicitly set to false, then the request is synchronous, we'll cover that a bit later. - user, password -- login and password for basic HTTP auth (if required). Please note that open call, contrary to its name, does not open the connection. It only configures the request, but the network activity only starts with the call of send. 3. Send it out. ```js xhr.send([body]) ``` This method opens the connection and sends the request to server. The optional body parameter contains the request body. Some request methods like GET do not have a body. And some of them like POST use body to send the data to the server. We'll see examples of that later. 4. Listen to xhr events for response. These three events are the most widely used: - load -- when the request is complete (even if HTTP status is like 400 or 500), and the response is fully downloaded. - error -- when the request couldn't be made, e.g. network down or invalid URL. - progress -- triggers periodically while the response is being downloaded, reports how much has been downloaded. ```js xhr.onload = function() { alert(Loaded: ${xhr.status} ${xhr.response}); xhr.onerror = function() { // only triggers if the request couldn't be made at all alert(Network Error); xhr.onprogress = function(event) { // triggers periodically // event.loaded - how many bytes downloaded // event.lengthComputable = true if the server sent Content-Length header // event.total - total number of bytes (if lengthComputable) alert(Received ${event.loaded} of ${event.total}); ``` Here's a full example. The code below loads the URL at /article/xmlhttprequest/example/load from the server and prints the progress: Once the server has responded, we can receive the result in the following xhr properties: status : HTTP status code (a number): 200, 404, 403 and so on, can be 0 in case of a non-HTTP failure. statusText : HTTP status message (a string): usually OK for 200, Not Found for 404, Forbidden for 403 and so on. response (old scripts may use responseText) : The server response body. We can also specify a timeout using the corresponding property: If the request does not succeed within the given time, it gets canceled and timeout event triggers. let url = new URL('https://google.com/search'); url.searchParams.set('q', 'test me!'); // the parameter 'q' is encoded xhr.open('GET', url); // https://google.com/search?q=test+me%21 ## Response Type We can use xhr.responseType property to set the response format: - "" (default) -- get as string, - "text" -- get as string, - "arraybuffer" -- get as ArrayBuffer (for binary data, see chapter <info:arraybuffer-binary-arrays>), - "blob" -- get as Blob (for binary data, see chapter <info:blob>), - "document" -- get as XML document (can use XPath and other XML methods) or HTML document (based on the MIME type of the received data), - "json" -- get as JSON (parsed automatically). For example, let's get the response as JSON: ## Ready states XMLHttpRequest changes between states as it progresses. The current state is accessible as xhr.readyState. All states, as in the specification: An XMLHttpRequest object travels them in the order 0 -> 1 -> 2 -> 3 -> ... -> 3 -> 4. State 3 repeats every time a data packet is received over the network. We can track them using readystatechange event: You can find readystatechange listeners in really old code, it's there for historical reasons, as there was a time when there were no load and other events. Nowadays, load/error/progress handlers deprecate it. ## Aborting request We can terminate the request at any time. The call to xhr.abort() does that: That triggers abort event, and xhr.status becomes 0. ## Synchronous requests If in the open method the third parameter async is set to false, the request is made synchronously. In other words, JavaScript execution pauses at send() and resumes when the response is received. Somewhat like alert or prompt commands. Here's the rewritten example, the 3rd parameter of open is false: It might look good, but synchronous calls are used rarely, because they block in-page JavaScript till the loading is complete. In some browsers it becomes impossible to scroll. If a synchronous call takes too much time, the browser may suggest to close the "hanging" webpage. Many advanced capabilities of XMLHttpRequest, like requesting from another domain or specifying a timeout, are unavailable for synchronous requests. Also, as you can see, no progress indication. Because of all that, synchronous requests are used very sparingly, almost never. We won't talk about them any more. ## HTTP-headers XMLHttpRequest allows both to send custom headers and read headers from the response. There are 3 methods for HTTP-headers: setRequestHeader(name, value) : Sets the request header with the given name and value. For instance: ```js xhr.setRequestHeader('Content-Type', 'application/json'); ``` ```warn header="Headers limitations" Several headers are managed exclusively by the browser, e.g. Referer and Host. The full list is in the specification-method). XMLHttpRequest is not allowed to change them, for the sake of user safety and correctness of the request. ``` ````warn header="Can't remove a header" Another peculiarity of XMLHttpRequest is that one can't undo setRequestHeader. Once the header is set, it's set. Additional calls add information to the header, don't overwrite it. For instance: ```js xhr.setRequestHeader('X-Auth', '123'); xhr.setRequestHeader('X-Auth', '456'); // the header will be: // X-Auth: 123, 456 ``` ```` getResponseHeader(name) : Gets the response header with the given name (except Set-Cookie and Set-Cookie2). For instance: ```js xhr.getResponseHeader('Content-Type') ``` getAllResponseHeaders() : Returns all response headers, except Set-Cookie and Set-Cookie2. Headers are returned as a single line, e.g.: ```http Cache-Control: max-age=31536000 Content-Length: 4260 Content-Type: image/png Date: Sat, 08 Sep 2012 16:53:16 GMT ``` The line break between headers is always "\ \ " (doesn't depend on OS), so we can easily split it into individual headers. The separator between the name and the value is always a colon followed by a space ": ". That's fixed in the specification. So, if we want to get an object with name/value pairs, we need to throw in a bit JS. Like this (assuming that if two headers have the same name, then the latter one overwrites the former one): ```js let headers = xhr .getAllResponseHeaders() .split('\ \ ') .reduce((result, current) => { let [name, value] = current.split(': '); result[name] = value; return result; }, {}); // headers['Content-Type'] = 'image/png' ``` ## POST, FormData To make a POST request, we can use the built-in FormData object. The syntax: We create it, optionally fill from a form, append more fields if needed, and then: 1. xhr.open('POST', ...) – use POST method. 2. xhr.send(formData) to submit the form to the server. For instance: The form is sent with multipart/form-data encoding. Or, if we like JSON more, then JSON.stringify and send as a string. Just don't forget to set the header Content-Type: application/json, many server-side frameworks automatically decode JSON with it: The .send(body) method is pretty omnivore. It can send almost any body, including Blob and BufferSource objects. ## Upload progress The progress event triggers only on the downloading stage. That is: if we POST something, XMLHttpRequest first uploads our data (the request body), then downloads the response. If we're uploading something big, then we're surely more interested in tracking the upload progress. But xhr.onprogress doesn't help here. There's another object, without methods, exclusively to track upload events: xhr.upload. It generates events, similar to xhr, but xhr.upload triggers them solely on uploading: - loadstart -- upload started. - progress -- triggers periodically during the upload. - abort -- upload aborted. - error -- non-HTTP error. - load -- upload finished successfully. - timeout -- upload timed out (if timeout property is set). - loadend -- upload finished with either success or error. Example of handlers: Here's a real-life example: file upload with progress indication: ## Cross-origin requests XMLHttpRequest can make cross-origin requests, using the same CORS policy as fetch. Just like fetch, it doesn't send cookies and HTTP-authorization to another origin by default. To enable them, set xhr.withCredentials to true: See the chapter <info:fetch-crossorigin> for details about cross-origin headers. ## Summary Typical code of the GET-request with XMLHttpRequest: There are actually more events, the modern specification lists them (in the lifecycle order): - loadstart -- the request has started. - progress -- a data packet of the response has arrived, the whole response body at the moment is in response. - abort -- the request was canceled by the call xhr.abort(). - error -- connection error has occurred, e.g. wrong domain name. Doesn't happen for HTTP-errors like 404. - load -- the request has finished successfully. - timeout -- the request was canceled due to timeout (only happens if it was set). - loadend -- triggers after load, error, timeout or abort. The error, abort, timeout, and load events are mutually exclusive. Only one of them may happen. The most used events are load completion (load), load failure (error), or we can use a single loadend handler and check the properties of the request object xhr to see what happened. We've already seen another event: readystatechange. Historically, it appeared long ago, before the specification settled. Nowadays, there's no need to use it, we can replace it with newer events, but it can often be found in older scripts. If we need to track uploading specifically, then we should listen to same events on xhr.upload object.
Resumable file upload
With fetch method it's fairly easy to upload a file. How to resume the upload after lost connection? There's no built-in option for that, but we have the pieces to implement it. Resumable uploads should come with upload progress indication, as we expect big files (if we may need to resume). So, as fetch doesn't allow to track upload progress, we'll use XMLHttpRequest. ## Not-so-useful progress event To resume upload, we need to know how much was uploaded till the connection was lost. There's xhr.upload.onprogress to track upload progress. Unfortunately, it won't help us to resume the upload here, as it triggers when the data is sent, but was it received by the server? The browser doesn't know. Maybe it was buffered by a local network proxy, or maybe the remote server process just died and couldn't process them, or it was just lost in the middle and didn't reach the receiver. That's why this event is only useful to show a nice progress bar. To resume upload, we need to know exactly the number of bytes received by the server. And only the server can tell that, so we'll make an additional request. ## Algorithm 1. First, create a file id, to uniquely identify the file we're going to upload: ```js let fileId = file.name + '-' + file.size + '-' + file.lastModified; ``` That's needed for resume upload, to tell the server what we're resuming. If the name or the size or the last modification date changes, then there'll be another fileId. 2. Send a request to the server, asking how many bytes it already has, like this: ```js let response = await fetch('status', { headers: { 'X-File-Id': fileId }); // The server has that many bytes let startByte = +await response.text(); ``` This assumes that the server tracks file uploads by X-File-Id header. Should be implemented at server-side. If the file doesn't yet exist at the server, then the server response should be 0 3. Then, we can use Blob method slice to send the file from startByte: ```js xhr.open("POST", "upload"); // File id, so that the server knows which file we upload xhr.setRequestHeader('X-File-Id', fileId); // The byte we're resuming from, so the server knows we're resuming xhr.setRequestHeader('X-Start-Byte', startByte); xhr.upload.onprogress = (e) => { console.log(Uploaded ${startByte + e.loaded} of ${startByte + e.total}); // file can be from input.files[0] or another source xhr.send(file.slice(startByte)); ``` Here we send the server both file id as X-File-Id, so it knows which file we're uploading, and the starting byte as X-Start-Byte, so it knows we're not uploading it initially, but resuming. The server should check its records, and if there was an upload of that file, and the current uploaded size is exactly X-Start-Byte, then append the data to it. Here's the demo with both client and server code, written on Node.js. It works only partially on this site, as Node.js is behind another server named Nginx, that buffers uploads, passing them to Node.js when fully complete. But you can download it and run locally for the full demonstration: [codetabs src="upload-resume" height=200] As we can see, modern networking methods are close to file managers in their capabilities -- control over headers, progress indicator, sending file parts, etc. We can implement resumable upload and much more.
Long polling
Long polling is the simplest way of having persistent connection with server, that doesn't use any specific protocol like WebSocket or Server Sent Events. Being very easy to implement, it's also good enough in a lot of cases. ## Regular Polling The simplest way to get new information from the server is periodic polling. That is, regular requests to the server: "Hello, I'm here, do you have any information for me?". For example, once every 10 seconds. In response, the server first takes a notice to itself that the client is online, and second - sends a packet of messages it got till that moment. That works, but there are downsides: 1. Messages are passed with a delay up to 10 seconds (between requests). 2. Even if there are no messages, the server is bombed with requests every 10 seconds, even if the user switched somewhere else or is asleep. That's quite a load to handle, speaking performance-wise. So, if we're talking about a very small service, the approach may be viable, but generally, it needs an improvement. ## Long polling So-called "long polling" is a much better way to poll the server. It's also very easy to implement, and delivers messages without delays. The flow: 1. A request is sent to the server. 2. The server doesn't close the connection until it has a message to send. 3. When a message appears - the server responds to the request with it. 4. The browser makes a new request immediately. This situation, where the browser has sent a request and keeps a pending connection with the server, is standard for this method. Only when a message is delivered, the connection is closed and reestablished. If the connection is lost, because of, say, a network error, the browser immediately sends a new request. A sketch of client-side subscribe function that makes long requests: As you can see, subscribe function makes a fetch, then waits for the response, handles it and calls itself again. ## Demo: a chat Here's a demo chat, you can also download it and run locally (if you're familiar with Node.js and can install modules): [codetabs src="longpoll" height=500] Browser code is in browser.js. ## Area of usage Long polling works great in situations when messages are rare. If messages come very often, then the chart of requesting-receiving messages, painted above, becomes saw-like. Every message is a separate request, supplied with headers, authentication overhead, and so on. So, in this case, another method is preferred, such as Websocket or Server Sent Events.
WebSocket
The WebSocket protocol, described in the specification RFC 6455, provides a way to exchange data between browser and server via a persistent connection. The data can be passed in both directions as "packets", without breaking the connection and the need of additional HTTP-requests. WebSocket is especially great for services that require continuous data exchange, e.g. online games, real-time trading systems and so on. ## A simple example To open a websocket connection, we need to create new WebSocket using the special protocol ws in the url: There's also encrypted wss:// protocol. It's like HTTPS for websockets. Once the socket is created, we should listen to events on it. There are totally 4 events: - open -- connection established, - message -- data received, - error -- websocket error, - close -- connection closed. ...And if we'd like to send something, then socket.send(data) will do that. Here's an example: For demo purposes, there's a small server server.js written in Node.js, for the example above, running. It responds with "Hello from server, John", then waits 5 seconds and closes the connection. So you'll see events open -> message -> close. That's actually it, we can talk WebSocket already. Quite simple, isn't it? Now let's talk more in-depth. ## Opening a websocket When new WebSocket(url) is created, it starts connecting immediately. During the connection, the browser (using headers) asks the server: "Do you support Websocket?" And if the server replies "yes", then the talk continues in WebSocket protocol, which is not HTTP at all. Here's an example of browser headers for a request made by new WebSocket("wss://javascript.info/chat"). - Origin -- the origin of the client page, e.g. https://javascript.info. WebSocket objects are cross-origin by nature. There are no special headers or other limitations. Old servers are unable to handle WebSocket anyway, so there are no compatibility issues. But the Origin header is important, as it allows the server to decide whether or not to talk WebSocket with this website. - Connection: Upgrade -- signals that the client would like to change the protocol. - Upgrade: websocket -- the requested protocol is "websocket". - Sec-WebSocket-Key -- a random browser-generated key, used to ensure that the server supports WebSocket protocol. It's random to prevent proxies from caching any following communication. - Sec-WebSocket-Version -- WebSocket protocol version, 13 is the current one. If the server agrees to switch to WebSocket, it should send code 101 response: Here Sec-WebSocket-Accept is Sec-WebSocket-Key, recoded using a special algorithm. Upon seeing it, the browser understands that the server really does support the WebSocket protocol. Afterwards, the data is transferred using the WebSocket protocol, we'll see its structure ("frames") soon. And that's not HTTP at all. ### Extensions and subprotocols There may be additional headers Sec-WebSocket-Extensions and Sec-WebSocket-Protocol that describe extensions and subprotocols. For instance: - Sec-WebSocket-Extensions: deflate-frame means that the browser supports data compression. An extension is something related to transferring the data, functionality that extends the WebSocket protocol. The header Sec-WebSocket-Extensions is sent automatically by the browser, with the list of all extensions it supports. - Sec-WebSocket-Protocol: soap, wamp means that we'd like to transfer not just any data, but the data in SOAP or WAMP ("The WebSocket Application Messaging Protocol") protocols. WebSocket subprotocols are registered in the IANA catalogue. So, this header describes the data formats that we're going to use. This optional header is set using the second parameter of new WebSocket. That's the array of subprotocols, e.g. if we'd like to use SOAP or WAMP: ```js let socket = new WebSocket("wss://javascript.info/chat", ["soap", "wamp"]); ``` The server should respond with a list of protocols and extensions that it agrees to use. For example, the request: Response: Here the server responds that it supports the extension "deflate-frame", and only SOAP of the requested subprotocols. ## Data transfer WebSocket communication consists of "frames" -- data fragments, that can be sent from either side, and can be of several kinds: - "text frames" -- contain text data that parties send to each other. - "binary data frames" -- contain binary data that parties send to each other. - "ping/pong frames" are used to check the connection, sent from the server, the browser responds to these automatically. - there's also "connection close frame" and a few other service frames. In the browser, we directly work only with text or binary frames. WebSocket .send() method can send either text or binary data. A call socket.send(body) allows body in string or a binary format, including Blob, ArrayBuffer, etc. No settings are required: just send it out in any format. When we receive the data, text always comes as string. And for binary data, we can choose between Blob and ArrayBuffer formats. That's set by socket.binaryType property, it's "blob" by default, so binary data comes as Blob objects. Blob is a high-level binary object, it directly integrates with <a>, <img> and other tags, so that's a sane default. But for binary processing, to access individual data bytes, we can change it to "arraybuffer": ## Rate limiting Imagine, our app is generating a lot of data to send. But the user has a slow network connection, maybe on a mobile internet, outside of a city. We can call socket.send(data) again and again. But the data will be buffered (stored) in memory and sent out only as fast as network speed allows. The socket.bufferedAmount property stores how many bytes remain buffered at this moment, waiting to be sent over the network. We can examine it to see whether the socket is actually available for transmission. ## Connection close Normally, when a party wants to close the connection (both browser and server have equal rights), they send a "connection close frame" with a numeric code and a textual reason. The method for that is: - code is a special WebSocket closing code (optional) - reason is a string that describes the reason of closing (optional) Then the other party in the close event handler gets the code and the reason, e.g.: Most common code values: - 1000 -- the default, normal closure (used if no code supplied), - 1006 -- no way to set such code manually, indicates that the connection was lost (no close frame). There are other codes like: - 1001 -- the party is going away, e.g. server is shutting down, or a browser leaves the page, - 1009 -- the message is too big to process, - 1011 -- unexpected error on server, - ...and so on. The full list can be found in RFC6455, §7.4.1. WebSocket codes are somewhat like HTTP codes, but different. In particular, codes lower than 1000 are reserved, there'll be an error if we try to set such a code. ## Connection state To get connection state, additionally there's socket.readyState property with values: - 0 -- "CONNECTING": the connection has not yet been established, - 1 -- "OPEN": communicating, - 2 -- "CLOSING": the connection is closing, - 3 -- "CLOSED": the connection is closed. ## Chat example Let's review a chat example using browser WebSocket API and Node.js WebSocket module <https://github.com/websockets/ws>. We'll pay the main attention to the client side, but the server is also simple. HTML: we need a <form> to send messages and a <div> for incoming messages: From JavaScript we want three things: 1. Open the connection. 2. On form submission -- socket.send(message) for the message. 3. On incoming message -- append it to div#messages. Here's the code: Server-side code is a little bit beyond our scope. Here we'll use Node.js, but you don't have to. Other platforms also have their means to work with WebSocket. The server-side algorithm will be: 1. Create clients = new Set() -- a set of sockets. 2. For each accepted websocket, add it to the set clients.add(socket) and set message event listener to get its messages. 3. When a message is received: iterate over clients and send it to everyone. 4. When a connection is closed: clients.delete(socket). Here's the working example: [iframe src="chat" height="100" zip] You can also download it (upper-right button in the iframe) and run it locally. Just don't forget to install Node.js and npm install ws before running. ## Summary WebSocket is a modern way to have persistent browser-server connections. - WebSockets don't have cross-origin limitations. - They are well-supported in browsers. - Can send/receive strings and binary data. The API is simple. Methods: - socket.send(data), - socket.close([code], [reason]). Events: - open, - message, - error, - close. WebSocket by itself does not include reconnection, authentication and many other high-level mechanisms. So there are client/server libraries for that, and it's also possible to implement these capabilities manually. Sometimes, to integrate WebSocket into existing projects, people run a WebSocket server in parallel with the main HTTP-server, and they share a single database. Requests to WebSocket use wss://ws.site.com, a subdomain that leads to the WebSocket server, while https://site.com goes to the main HTTP-server. Surely, other ways of integration are also possible.
Server Sent Events
The Server-Sent Events specification describes a built-in class EventSource, that keeps connection with the server and allows to receive events from it. Similar to WebSocket, the connection is persistent. But there are several important differences: EventSource is a less-powerful way of communicating with the server than WebSocket. Why should one ever use it? The main reason: it's simpler. In many applications, the power of WebSocket is a little bit too much. We need to receive a stream of data from server: maybe chat messages or market prices, or whatever. That's what EventSource is good at. Also it supports auto-reconnect, something we need to implement manually with WebSocket. Besides, it's a plain old HTTP, not a new protocol. ## Getting messages To start receiving messages, we just need to create new EventSource(url). The browser will connect to url and keep the connection open, waiting for events. The server should respond with status 200 and the header Content-Type: text/event-stream, then keep the connection and write messages into it in the special format, like this: - A message text goes after data:, the space after the colon is optional. - Messages are delimited with double line breaks \ \ . - To send a line break \ , we can immediately send one more data: (3rd message above). In practice, complex messages are usually sent JSON-encoded. Line-breaks are encoded as \ within them, so multiline data: messages are not necessary. For instance: ...So we can assume that one data: holds exactly one message. For each such message, the message event is generated: ### Cross-origin requests EventSource supports cross-origin requests, like fetch and any other networking methods. We can use any URL: The remote server will get the Origin header and must respond with Access-Control-Allow-Origin to proceed. To pass credentials, we should set the additional option withCredentials, like this: Please see the chapter <info:fetch-crossorigin> for more details about cross-origin headers. ## Reconnection Upon creation, new EventSource connects to the server, and if the connection is broken -- reconnects. That's very convenient, as we don't have to care about it. There's a small delay between reconnections, a few seconds by default. The server can set the recommended delay using retry: in response (in milliseconds): The retry: may come both together with some data, or as a standalone message. The browser should wait that many milliseconds before reconnecting. Or longer, e.g. if the browser knows (from OS) that there's no network connection at the moment, it may wait until the connection appears, and then retry. - If the server wants the browser to stop reconnecting, it should respond with HTTP status 204. - If the browser wants to close the connection, it should call eventSource.close(): Also, there will be no reconnection if the response has an incorrect Content-Type or its HTTP status differs from 301, 307, 200 and 204. In such cases the "error" event will be emitted, and the browser won't reconnect. ## Message id When a connection breaks due to network problems, either side can't be sure which messages were received, and which weren't. To correctly resume the connection, each message should have an id field, like this: When a message with id: is received, the browser: - Sets the property eventSource.lastEventId to its value. - Upon reconnection sends the header Last-Event-ID with that id, so that the server may re-send following messages. ## Connection status: readyState The EventSource object has readyState property, that has one of three values: When an object is created, or the connection is down, it's always EventSource.CONNECTING (equals 0). We can query this property to know the state of EventSource. ## Event types By default EventSource object generates three events: - message -- a message received, available as event.data. - open -- the connection is open. - error -- the connection could not be established, e.g. the server returned HTTP 500 status. The server may specify another type of event with event: ... at the event start. For example: To handle custom events, we must use addEventListener, not onmessage: ## Full example Here's the server that sends messages with 1, 2, 3, then bye and breaks the connection. Then the browser automatically reconnects. [codetabs src="eventsource"] ## Summary EventSource object automatically establishes a persistent connection and allows the server to send messages over it. It offers: - Automatic reconnect, with tunable retry timeout. - Message ids to resume events, the last received identifier is sent in Last-Event-ID header upon reconnection. - The current state is in the readyState property. That makes EventSource a viable alternative to WebSocket, as the latter is more low-level and lacks such built-in features (though they can be implemented). In many real-life applications, the power of EventSource is just enough. Supported in all modern browsers (not IE). The syntax is: The second argument has only one possible option: { withCredentials: true }, it allows sending cross-origin credentials. Overall cross-origin security is same as for fetch and other network methods. ### Properties of an EventSource object readyState : The current connection state: either EventSource.CONNECTING (=0), EventSource.OPEN (=1) or EventSource.CLOSED (=2). lastEventId : The last received id. Upon reconnection the browser sends it in the header Last-Event-ID. ### Methods close() : Closes the connection. ### Events message : Message received, the data is in event.data. open : The connection is established. error : In case of an error, including both lost connection (will auto-reconnect) and fatal errors. We can check readyState to see if the reconnection is being attempted. The server may set a custom event name in event:. Such events should be handled using addEventListener, not on<event>. ### Server response format The server sends messages, delimited by \ \ . A message may have following fields: - data: -- message body, a sequence of multiple data is interpreted as a single message, with \ between the parts. - id: -- renews lastEventId, sent in Last-Event-ID on reconnect. - retry: -- recommends a retry delay for reconnections in ms. There's no way to set it from JavaScript. - event: -- event name, must precede data:. A message may include one or more fields in any order, but id: usually goes the last.