January 30, 2015, 5:02 pm
Hey! I have been trying to set up a Proxmox VE cluster for the past couple of days and I have encountered some problems.
First of all I would like to explain that my hosting provider does not support Multicast, if I could use it I would but they tell me that it isn't possible.
So, I currently have followed the following guide:
https://pve.proxmox.com/wiki/Proxmox_VE_2.0_Cluster
In addition I also set up Unicast as instructed here:
https://pve.proxmox.com/wiki/Multica..._else_fails.29
Lastly I started on this guide for setting up a Two-Node HA cluster:
https://pve.proxmox.com/wiki/Two-Nod...BD_split-brain
And got as far as setting up two_node and expected_votes in cman. I saved cluster.conf.new, verified it and applied it in the control panel where both of my nodes are visible (although one is currently in read-only mode because of expected votes being 2). After a reboot it appears that the cluster.conf file has only been updated on Node1, not on Node2. Ontop of this I can't access anything on Node2, I can't create VM's or CT's on it and I can't access it's file storage.
Here are some pictures:
![]()
![]()
![]()
My hosts files are set up, the servers can ping eachother through the node names.
Does anyone know what the problem is? Is there any way I can force Node2 to update its cluster.conf?
↧
January 31, 2015, 2:43 am
Yesterday my OVH-vRack goes offline! (it's a verry stupid network design there also!)
Cet équipement ne répond plus. Nous investiguons. <- if happen this will hurt u verry much!
Ceph and Proxmox stops working and 2 vm's are crashing, after the vRack comes back
ceph do the resync and everything looks good for the first moment! (was verry happy not losing data (replica 3))
But proxmox starts hurting me badly! Cluster runs out of control and node1 was fencing the two other nodes even ceph was online! The rgmanager have problems to connect each other. I have to restart all 3 nodes! There was no HA working, i was not able to start, stop and migrate vm's manualy! :-(
rgmanager doesn't stop on the nodes (wait 5 minutes) so i have to killall rgmanger tasks, then the nodes are rebooting.
Now all 3 nodes are online but on node3 i see this in boot.log:
Fri Jan 30 18:31:43 2015: Starting cluster: Fri Jan 30 18:31:43 2015: Checking if cluster has been disabled at boot... [ OK ]
Fri Jan 30 18:31:43 2015: Checking Network Manager... [ OK ]
Fri Jan 30 18:31:43 2015: Global setup... [ OK ]
Fri Jan 30 18:31:43 2015: Loading kernel modules... [ OK ]
Fri Jan 30 18:31:43 2015: Mounting configfs... [ OK ]
Fri Jan 30 18:31:43 2015: Starting cman... [ OK ]
Fri Jan 30 18:31:52 2015: Waiting for quorum... Timed-out waiting for cluster
Fri Jan 30 18:32:30 2015: [FAILED]
....
Fri Jan 30 18:32:42 2015: cluster not ready - no quorum?
Cluster Status for cRZ @ Sat Jan 31 11:13:31 2015
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
node1pv 1 Online
node2pv 2 Online
node3pv 3 Online, Local
Version: 6.2.0
Config Version: 14465
Cluster Name: cRZ
Cluster Id: 3710
Cluster Member: Yes
Cluster Generation: 76348
Membership state: Cluster-Member
Nodes: 3
Expected votes: 3
Total votes: 3
Node votes: 1
Quorum: 2
Active subsystems: 1
Flags:
Ports Bound: 0
Node name: node3pv
Node ID: 3
Multicast addresses: 239.192.14.140
Node addresses: 10.10.10.3
root@node1pv:~# pvecm nodes
Node Sts Inc Joined Name
1 M 76320 2015-01-30 17:48:14 node1pv
2 M 76348 2015-01-30 18:40:07 node2pv
3 M 76344 2015-01-30 18:31:41 node3pv
root@node2pv:~# pvecm nodes
Node Sts Inc Joined Name
1 M 76348 2015-01-30 18:40:07 node1pv
2 M 76340 2015-01-30 18:39:40 node2pv
3 M 76348 2015-01-30 18:40:07 node3pv
root@node3pv:~# pvecm nodes
Node Sts Inc Joined Name
1 M 76348 2015-01-30 18:40:07 node1pv
2 M 76348 2015-01-30 18:40:07 node2pv
3 M 76336 2015-01-30 18:31:41 node3pv
Now my questions:
So there is no fault tolerance in proxmox for such providers with a lack of HA infrastucture?
Now is it running or not, looks like but i'm not sure!
How can i test the quorum is working on all 3 nodes?
Why every node has different times for joining?
Can proxmox communicate over different networks, is there a fallback option?
Because the "outside" network was still working!
Regards
↧
↧
January 31, 2015, 4:20 am
Hello! I use
https://github.com/CpuID/pve2-api-php-client client for proxmox and trying get RRD data:
PHP Code:
if ($px->constructor_success()) {
if ($px->login()) {
$px->set_debug(true);
$params = array();
$params['timeframe'] = "day";
$params['ds'] = "cpu";
$params['cf'] = "MAX";
$png = $px->get("/nodes/node/qemu/100/rrd",$params);
print_r($png);
}
}
and debug error:
PHP Code:
FULL RESPONSE:HTTP/1.1 400 Parameter verification failed.
Cache-Control: max-age=0Connection: closeDate: Sat, 31 Jan 2015 12:25:11 GMT
Pragma: no-cacheServer: pve-api-daemon/3.0Content-Length: 131Content-Type: application/json;charset=UTF-8
Expires: Sat, 31 Jan 2015 12:25:11 GMT{"errors":{"timeframe":"property is missing and it is not optional","ds":"property is missing and it is not optional"},"data":null}END
FULL RESPONSE.Headers:HTTP/1.1 400 Parameter verification failed.Cache-Control: max-age=0Connection: closeDate: Sat, 31 Jan 2015 12:25:11 GMTPragma: no-cacheServer: pve-api-daemon/3.0Content-Length: 131Content-Type: application/json;charset=UTF-8Expires: Sat, 31 Jan 2015 12:25:11 GMTEnd Headers.Data:{"errors":{"timeframe":"property is missing and it is not optional","ds":"property is missing and it is not optional"},"data":null}End Headers.RESPONSE ARRAY:Array( [errors] => Array ( [timeframe] => property is missing and it is not optional [ds] => property is missing and it is not optional ) [data] => )END RESPONSE ARRAY.----------------------------------------------This API Request Failed.HTTP Response - 400HTTP Error - HTTP/1.1 400 Parameter verification failed.
In ssh:
pvesh get /nodes/{node}/qemu/{vmid}/rrd - job and get array data.
What am I doing wrong?
↧
January 31, 2015, 6:57 am
Hello,
I just ask the forum hoping to find an answer to my problem.
So I installed a Proxmox 3.3 virtualization server and I can not ping my virtual machines when I configured them in connection bridge.
For example, I create a virtual machine in Windows 7 I configured connection bridge with a manual IP address and I just can not ping my machines on the network from the latter (not even the Proxmox server).
Also, when I configure the virtual machine NAT connection, I get no problems accessing the Internet (of course I do not see other machines on the network which is normal).
If someone would have an idea, I would be very grateful.
Best Regards.
↧
January 31, 2015, 7:09 am
Wondering if it is somehow possible from a CLI of a PVE node to connect to a running VM's serial console like with libvirt and virsh console?
↧
↧
January 31, 2015, 1:15 pm
I'm using mdadm raid10 with 8x 512gb Crucial SSD.
root@server:~# pveperf
CPU BOGOMIPS: 48002.04
REGEX/SECOND: 765111
HD SIZE: 19.69 GB (/dev/mapper/pve-root)
BUFFERED READS: 585.00 MB/sec
AVERAGE SEEK TIME: 0.22 ms
FSYNCS/SECOND: 111.79
DNS EXT: 84.20 ms
DNS INT: 3.57 ms
Another time, I see a post here with someone saying FSYNCS/SECOND is good when we have value of 1000+
OMG, what is the problem? It's 8x SSDs.
Thank you guys for try to help-me!
@UDO, where is you? I love you!
↧
January 31, 2015, 1:47 pm
Looking for something like vCenter that we can add nodes and view all of them from one dashboard or the closest alternative.
Please advise..
Thanks,
↧
January 31, 2015, 3:35 pm
Hi,
I'm trying to mount a folder on host in one virtual Container server. I have created the file /etc/vz/conf/109.mount and put this inside:
#!/bin/bash
. /etc/vz/vz.conf
. ${VE_CONFFILE}
SRC=/mnt/disk01
DST=/mnt/disk01
if [ ! -e ${VE_ROOT}${DST} ]; then mkdir -p ${VE_ROOT}${DST}; fi
mount -n -t simfs ${SRC} ${VE_ROOT}${DST} -o ${SRC}
the I try to do it executible:
chmod +x /etc/vz/conf/109.mount
but I only get "chmod: changing permissions of `/etc/pve/local/openvz/109.mount': Function not implemented".
How can I solve this?
I'm using Proxmox 3.3.
↧
February 1, 2015, 5:24 am
Hello,
After a fresh proxmox installation , a kvm VPS is shutting down by itself.
which logs do i have to monitor in order to debug this issue?
pve-manager/3.3-5/bfebec03 (running kernel: 2.6.32-34-pve)
↧
↧
February 1, 2015, 10:36 am
Hello,
I'm practically out of disk space on a Proxmox 2.x installation and I'm trying to backup the VMs on it onto a remote server. I attached a partition on the remote server to the Proxmox server with sshfs.
But everytime I run vzdump I get the message:
Quote:
vbed2:~# vzdump 102 -dumpdir remote-drive/INFO: starting new backup job: vzdump 102 --dumpdir remote-drive/
INFO: Starting Backup of VM 102 (qemu)
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: creating archive 'remote-drive/vzdump-qemu-102-2015_02_01-19_15_29.vma'
INFO: starting kvm to execute backup task
ERROR: unable to read file 'remote-drive/vzdump-qemu-102-2015_02_01-19_15_29.tmp/qemu-server.conf'
INFO: aborting backup job
INFO: stopping kvm after backup task
ERROR: Backup of VM 102 failed - unable to read file 'remote-drive/vzdump-qemu-102-2015_02_01-19_15_29.tmp/qemu-server.conf'
INFO: Backup job finished with errors
This applies to all VMs on the server. I've checked online and saw a similar situation mentioned in the mailing list (
http://pve.proxmox.com/pipermail/pve...er/006129.html), but no solution was actually offered.
Could I have some suggestions on how to get these VMs backed-up or migrated to another server please? Thank you.
↧
February 1, 2015, 11:44 am
Hi all,
I've been designing my home network for a while now and am starting to have a good idea on what I want in it. The idea is to have three main servers consisting of Sophos UTM, ProxMox, and UnRAID (all rack mounted with 24 bay Norco-4224 for UnRAID). As far as ProxMox goes I don't need any backups for the drives that store the VMs. I will offload all storage to the unraid server which will be configured in a way to make best use of a failure (such as two drives dedicated to VM backups instead of a giant 20 drive array). For now I plan to run a 10 or less Minecraft server for fooling around in on the side and possibly share out a VM or so to a friend. The rest will be whatever comes to mind at the same, though this will be my performance machine, so I want it to last a while. For now I have the rack and firewall running.
The current options I am thinking of are:
1: Using a single 500GB Samsung 850 EVO
2: Using a Perc 6/i or 6/E card with two 256GB Samsung 850 EVO drives in either raid 0 or 1.
3: Using something like a DS214+ with two 256GB Samsung 850 EVO drives in either raid 0 or 1 with one or both RJ45 ports plugged directly into the server.
Being that Minecraft will be one of the applications I'd like the storage to have a decent amount of IOPS. There will be no HDDs in this either since storage is handled by the unraid server, so this is specifically performance-based for the VMs. Also note I will be using an board with the C204 chipset and ECC ram. I can give exact specs if requested.
Overall what would be everyone's opinion on the storage option for such a server? And I do realize there will be a bunch of single points of failures in my network. I am not a datacenter so it is what it is. Best I can aim for in a home environment is good enough.
↧
February 1, 2015, 2:00 pm
Hi !
For some times I have seen posts related Qemu guest agent.
I think the native support is strategic (for Windows VM and VSS stabilization).
OK, I think that a Windows server is "sensitive", I do not doubt a Linux/BSD server is tollerant to a snapshot backup (state not stable) but with a Windows server (Exchange, and every devilry) VSS stabilization is absolutely necessary.
Any good news ??
Thanks
Luca
↧
February 1, 2015, 2:43 pm
Hello,
I've run Proxmox for a while, but now i need to put it in a datacentre, i curious whether I need to do any additional security in apache, i have read some posts saying that you can "deny all" in apache, use ssh keys etc - and im happy with all that - but what would be considered best practice. are most people running and separate firewall ? or are most people happy to run with the web interface internet facing.
Thanks in advance.
Cam
↧
↧
February 1, 2015, 7:54 pm
I want to know about RAID in ProLiant DL320e Gen8 V2 have support in RAID Proxmox V3.2 or not?
If not support, what module must install in this server.
Thans
↧
February 2, 2015, 3:52 am
Hi all,
I'm building a two node HA cluster and while testing I've noticed that the output of clustat never changes while my iscsi quorum disk is being rebooted:
Code:
Member Name ID Status ------ ---- ---- ------
pve1 1 Online, rgmanager
pve2 2 Online, Local, rgmanager
/dev/block/8:33 0 Online, Quorum Disk
Is this something to be concerned about? Clustat changes its output while one of the cluster nodes is rebooting
Thanks
NTB
↧
February 2, 2015, 6:01 am
Hello everebody,
I want to buy one upc for my dell server working with proxmox 3.3. Acording to workmates, APC Smart is the best upc(?) for linux debian server. I read there are 2 ways to manage upc from linux server trought usp port: nut and apcupsd. I don't have any experience about upc s and its management.
What kind of upc model do you recommend to buy that works fine with proxmox?
What is the best way to manage one UPS under proxmox: nut or apcupsd?
↧
February 2, 2015, 6:45 am
Hello,
Proxmox VE lacks on traffic accounting support and the ability to limit OpenVZ Containers.
That's why I thought, go with SolusVM or maybe you can do that yourself? Why has Proxmox not already done so?
I love OpenSource Projects so I stood with Proxmox.
So I developed my own Traffic Accounting and Limitation solution.
Features:
- KVM Traffic Accounting(NAT mode not tested, may not work)
- OpenVZ Traffic Accounting(Bridged mode not tested, may not work)
- KVM Limitation on the fly(Change Network speed while KVM is running)
- OpenVZ Network Limitation(You can set a network speed rate as you already know it from VMs(KVM))
- Automatically reset the traffic on a given date to 0(e.g. every month on the 15th) - Date can be set
- Automatically limit the VM/Container when it exceeds a defined bandwidth limit (e.g. Normal Network speed: 125MB/s ; After exceeding 1TB traffic limit to 20MB/s)
All these features are integrated with WHMCS. I only developed the API part. You can not administrate these settings via the Proxmox GUI.
There were some traffic calculation bugs, but seems to be fixed. I did not really stress tested it.
Currently I'm just taking the netin and netout values of the container, storing them in the pmxcfs filesystem and calculating traffic every 5 minutes.
Also every 5 minutes new limits will be applied.
So, every 5 minutes, for each VM a /etc/pve/[openvz/qemu-server]/VMID.conf like file will be read and written. Seems to work fine and doesn't lead to problems on a 500 VMs cluster.
For now, I won't make the code public accessibly because it could lead to more confusion. My Git Repository is not compatible with the official Proxmox branch, because I also implemented many other functions like:
- Reinstall an OpenVZ Container (Integrated in GUI)
- ISO Upload for customers(KVM) / also integrated into WHMCS
- Set CPU Limit for KVM and OpenVZ Containers (e.g. Host CPU has 3.6 Ghz and you can limit the vCores of a VM/CT to 800 Mhz)
- NoVNC Console as a own Service - Currently running on Port 3129 so we can block the GUI (8006) for customers and use WHMCS for administration
- TUN/TAP can be enabled/disabled in an OpenVZ Container via API
- FUSE can be enabled/disabled in an OpenVZ Container via API
- Root password of an OpenVZ Container can be changed via API
- Fixed: extended attributes of an OpenVZ Container are set(i) and so the files cannot be deleted / You first need to do a chattr -ia -R /var/lib/vz/private/... before removing that container
I think Traffic accounting is an important feature that is really wanted by users of Proxmox VE and i don't understand why you didn't implemented that yet. Hope to get a response of a Proxmox developer about that.
If anyone would give some advice, would like to discuss some more things or review code, I would be very thankful.
Best regards
Henry
↧
↧
February 2, 2015, 10:15 am
Hi,
I have Proxmox 3.2-4 on cluster mode and have multiple CTs or Ubuntu 12.04 working on it. I have FreeNAS NFS storage and all the CTs to NFS Storage. I have no issues with the system. I am able to migrate the CT from one server to another server and everything is working well.
Recently we wanted to install new application hence we need to install Ubuntu 14.04 LTS. I created same CT on the NFS storage but we have issues with installation of CT. I downloaded the images from different sources Openvz.com, image from another server and its not working properly. We have below issue:
root@server:~# cd ~
root@server:~# touch test.txt
root@server:~# lsattr
lsattr: Inappropriate ioctl for device While reading flags on ./test.txt
root@server:~#
I have tried debian as well but still not working properly. Not sure if the issue is related to FreeNAS NFS but i have similar setup at two locations and i have the same issue at both the location.
Any help on this is much appreciated.
Thanks
↧
February 2, 2015, 11:20 am
Hello,
i've setuped this day an machine with the new Proxmoxinstaller. I build the system with ZFS Raid1. So i would like to to simulate a harddisk failure. Ok, changed one disk:
Code:
pool: rpool state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-4J
scan: none requested
config:
NAME STATE READ WRITE CKSUM
rpool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
sda3 ONLINE 0 0 0
sdb3 UNAVAIL 0 116 0 corrupted data
So the first thing was, to put a new disk in server, done... but... nothing is going. The system can't see disk. So google around and found this:
Code:
echo "0 0 0" >/sys/class/scsi_host/host3/scan
Scan every port. But i think there is a special command for ZFS. I Hope So. Then i can see the disk with fdisk.
Code:
Disk /dev/sdb: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders, total 488397168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xbe36be36
Device Boot Start End Blocks Id System
/dev/sdb1 * 2048 718847 358400 7 HPFS/NTFS/exFAT
/dev/sdb2 718848 488394751 243837952 7 HPFS/NTFS/exFAT
Ok, so i would like to replace the disk. First trial:
Code:
zpool replace -f rpool /dev/sdb3 /dev/sdb3
cannot open '/dev/sdb3': No such device or address
cannot replace /dev/sdb3 with /dev/sdb3: one or more devices is currently unavailable
Second trail:
Code:
zpool replace -f rpool /dev/sdb3 /dev/sdb
Make sure to wait until resilver is done before rebooting.
First status:
Code:
pool: rpool
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Mon Feb 2 20:04:37 2015
371M scanned out of 639M at 11.2M/s, 0h0m to go
370M resilvered, 58.05% done
config:
NAME STATE READ WRITE CKSUM
rpool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
sda3 ONLINE 0 0 0
replacing-1 UNAVAIL 0 0 0
sdb3 UNAVAIL 0 116 0 corrupted data
sdb ONLINE 0 0 0 (resilvering)
After some minutes:
Code:
pool: rpool
state: ONLINE
scan: resilvered 639M in 0h0m with 0 errors on Mon Feb 2 20:05:31 2015
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sda3 ONLINE 0 0 0
sdb ONLINE 0 0 0
errors: No known data errors
Is this really ok? What is with EFI and boot listed with parted?
Code:
Model: ATA ST380817AS (scsi)
Disk /dev/sda: 80.0GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number Start End Size File system Name Flags
1 1049kB 2097kB 1049kB Grub-Boot-Partition bios_grub
2 2097kB 136MB 134MB fat32 EFI-System-Partition boot
3 136MB 80.0GB 79.9GB zfs PVE-ZFS-Partition
Model: ATA ST3250820NS (scsi)
Disk /dev/sdb: 250GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number Start End Size File system Name Flags
1 1049kB 250GB 250GB zfs zfs
9 250GB 250GB 8389kB
Model: Unknown (unknown)
Disk /dev/zd0: 4295MB
Sector size (logical/physical): 512B/4096B
Partition Table: loop
Number Start End Size File system Flags
1 0.00B 4295MB 4295MB linux-swap(v1)
So i think the system boots only with the first disk.
Thanks and best regards.
↧
February 2, 2015, 2:30 pm
Whom should I report it to?
↧
