We deployed the cluster using this template.We did a restart of the Client node VM; after the restart the elasticsearch service fails to start.Looking in the logs indicated that elasticsearch:elasticsearch did not have access to /datadisks/disk1.Doing a ls -l shows:dr-r-r- 4 root root 4096 Feb 17 23:04 disk1Tried to change permissions/ownership, and remount the drive to other points and same with rw,user,noatime with no change.We aren't by any means decent with.nix over here, but I wouldn't think a reboot would take a node down like this. Any help would be greatly appreciated. Here is more info from our investigations:We did a little more recon on this problem and found a number of issues:.The client nodes didn't survive a reboot because of a permissioning problem in the primary mount point that holds the ES data directory /datadisks/disk1.

• There Are No Ingest Nodes In This Cluster Unable To Forward Request To An Ingest Node

• Creating your cluster and configuring the quorum: Node and File Share Majority Introduction Welcome to Part 1 of my series 'Step-by-Step: Configuring a 2-node multi-site cluster on Windows Server 2008 R2'. Before we jump right in to the details, let's take a moment to discuss what exactly a multi-site cluster is and why I would.
• Dec 11, 2018  Caused by: java.lang.IllegalStateException: There are no ingest nodes in this cluster, unable to forward request to an ingest node. We should document this requirement for Monitoring in the appropriate place(s), even though by default all ES nodes are Ingest nodes.

The client node VMs were configured by the azure marketplace template to have no additional data disks attached. The custom VM configuration script '/src/scripts/vm-disk-utils-0.1.sh' then created an additional mount point against the temporary disk and placed the data directory there.

For some reason a reboot seems to have lowered the permission level of that mount. We think this can be fixed by editing the fstab entry to change ownership of the mount to the 'elasticsearch' user.

There Are No Ingest Nodes In This Cluster Unable To Forward Request To An Ingest Node

There are no ingest nodes in this cluster, unable to forward request to an ingest node. You need to make sure you have a node that can run the ingest pipelines.

But we're concerned that we may not want to place the ES data folder on the temporary disk. We need to confirm whether this should go onto the temporary disk or onto the permanent disk holding the root partition. This would affect any VMs that did not have additional data disks attached (so master and client nodes).The data nodes have multiple problems:.The azure marketplace script output shows that the correct data disks were detected and partitioned. It seems to show that they were then striped via mdadm correctly. However mdadm is now showing that only one of the two available partitions (/dev/sdd1) was placed into the RAID array.The azure marketplace script output shows that no fstab entry was created for the RAID array md0 since the UUID seems to have been found in the fstab at the time the script ran. Currently, there is no entry for the RAID array in /etc/fstab and there is no reason to suspect that a prior entry was there when vm-disk-utils-0.1.sh ran.

Also, because of a lack of fstab entry, the data nodes would not survive reboot (.caveat that they actually would in the current case because of the next bug).The RAID array md0 was never mounted correctly the first time (despite the script attempting to mount it to /datadisks/disk1). This problem went undetected by the azure marketplace script and it continued to write the ES data directory to the root partition. The end result is that the 1TB of additional attached ssds are being completely ignored by the cluster and all data is being written to a small drive where the root is mounted. Sometimes mdadm silently fails when creating RAID array on Ubuntu 16.04.

I have couple of Elasticsearch questions regarding client node:.Can I say: any nodes as long as they are opening HTTP port, I can treat them as 'client' nodes, because we can do search/index through this node.Actually we treat the node as client node when the cluster=false and data=false, if I set up 10 client nodes, do I need to route in my client side, I mean if I specify clientOne:9200 in my code as ES portal, then would clientOne forward other HTTP requests to other client nodes, otherwise, clientOne would be under very high pressure. I.e do they communicate with each other between client nodes?.When I specify client nodes in ES cluster, should I close other nodes' HTTP port?