Corporate Home Open Source Home
Syndicate content
Eucalyptus

Join us at engage.eucalyptus.com

2 replies [Last post]
reuben_rajesh
Offline
Joined: 10/13/2011

I have a setup a walrus service. I used CloudBerry explorer to browse the walrus account. I'm able to create bucket/add files using cloud berry to walrus. I have a written code earlier to communicate to Amazon cloud to upload files into my bucket using S3. This code is written in c#.net. Since walrus also supports AWSSDK, i tried to change the url to walrus service url and tried to access it, but it gave me error. I tried different ways but all in vain. I searched internet alot but there was no proper solution to access walrus client using c#.net. I have posted the C# code for listing the bucket here,

static string accessKey = "KxSyw67T0wVUvcZoGRMo07DTTSNXVoCpPTF5Q";
static string secretKey = "7Y92fV2q5yyOP4nsveQeyyEwSEl98RkZiuEpXQ";
static string url = "http://10.0.4.131:8773/services/Walrus";

public static void SimpleListBuckets(string awsId, string secretId)
{
try
{
// here is the basic Http Web Request
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
request.Method = "GET";
WebHeaderCollection headers = (request as HttpWebRequest).Headers;
// the canonical string combines the request's data
// with the current time
string httpDate = DateTime.UtcNow.ToString("ddd, dd MMM yyyy HH:mm:ss ") + "GMT";
headers.Add("x-amz-date", httpDate);
// our request is very simple, so we can hard-code the string
string canonicalString = "GET\n\n\n\nx-amz-date:" + httpDate + "\n/";
// now encode the canonical string
Encoding ae = new UTF8Encoding();
// create a hashing object
HMACSHA1 signature = new HMACSHA1();
// secretId is the hash key
signature.Key = ae.GetBytes(secretId);
byte[] bytes = ae.GetBytes(canonicalString);
byte[] moreBytes = signature.ComputeHash(bytes);
// convert the hash byte array into a base64 encoding
string encodedCanonical = Convert.ToBase64String(moreBytes);
// finally, this is the Authorization header.
headers.Add("Authorization", "AWS " + awsId + ":" + encodedCanonical);
HttpWebResponse response;
response = request.GetResponse() as HttpWebResponse;
// read the response stream and put it into a byte array
Stream stream = response.GetResponseStream() as Stream;
byte[] buffer = new byte[32 * 1024];
int nRead = 0;
MemoryStream ms = new MemoryStream();
do
{
nRead = stream.Read(buffer, 0, buffer.Length);
ms.Write(buffer, 0, nRead);
} while (nRead > 0);
// convert read bytes into string
ASCIIEncoding encoding = new ASCIIEncoding();
string responseString = encoding.GetString(ms.ToArray());
System.Console.WriteLine(responseString);
}
catch (Exception ex)
{
System.Console.WriteLine(ex.Message);
}
}

It is often giving error in the request.GetResponse() section. It is giving 404 bad request error, If i change the 'Walrus' to 'walrus' in the url, it is giving 500 internal error.

Is there a proper solution for this? Please help.

sang-min
Offline
Joined: 07/19/2010
Hi reuben, there is not an

Hi reuben,

there is not an easy way to solve this problem. In theory, both Walrus and your c# client speaks to the S3 standard. In reality, there are many ways that the implementation can go wrong. One way to debug this kind of problem is to compare wire messages between valid request-response and the one generated from your client. You may use one of the tools known to work with Walrus (http://open.eucalyptus.com/wiki/s3-compatible-tools) to generate the valid message.
you can use tcpdump to inspect the wire message (tcpdump -s0 -A -l 'tcp port 8773'). Then you should change your c# client to generate the same http request as the valid one. Cheers!

reuben_rajesh
Offline
Joined: 10/13/2011
Eucalyptus list bucket 1000 limitation

Hi,

Sorry for the delayed response and really thank you for your valuable comment. I have already figured out the the reason for the error. The actual reason was because of the change in the HTTP Header. The header object sent to Amazon is different and is not accepted by Walrus. Eg: 'x-amz-date' header in Amazon request. This i have fixed and is working fine.

Now I'm facing a different problem. If the Bucket has more than 1000 objects, then if we try to list objects in the particular bucket, S3 does return only 1000 object entries at a time. But the if the result is truncated, i.e., if all the objects are not returned the a flag <IsTruncated>true</IsTruncated> is set in the response. To get the remaining object entries, amazon suggests to set the NextMarker field of the next request to the <Marker> tag that we got from the response. The link is here! . This will be fine for Amazon bucket. But in case of Walrus bucket, if the object entries are more than 1000 and if we call ListBucketObjects, it is returning the first 1000 entries and 'IsTruncated' node is set to true. But the 'Marker' node is not present in the Response xml, without which i cannot query the next object entries. Do you have any idea how to resolve this?

Below is the sample response for Listing Amazon bucket,

<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Name>quotes</Name>
  <Prefix>N</Prefix>
  <Marker>Ned</Marker>
  <MaxKeys>40</MaxKeys>
  <IsTruncated>false;</IsTruncated>
  <Delimiter>/</Delimiter>
  <Contents>
    <Key>Nelson</Key>
    <LastModified>2005-11-17T07:13:48Z</LastModified>
    <ETag>&quot;828ef3fdfa96f00ad9f27c383fc9ac7f&quot;</ETag>
    <Size>5</Size>
    <StorageClass>STANDARD</StorageClass>
    <Owner>
      <ID>bcaf1ffd86f41caff1a493dc2ad8c2c281e37522a640e161ca5fb16fd081034f</ID>
      <DisplayName>webfile</DisplayName>
     </Owner>
  </Contents>
   ...    
   ... 
</ListBucketResult>

Below is the actual walrus response for listing bucket objects,

<ListBucketResponse xmlns="http://s3.amazonaws.com/doc/2006-03-01/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ListBucketResponse>
    <MetaData>
      <Name>lastwritetime</Name>
      <Value>11/16/2011 5:07:42 PM</Value>
    </MetaData>
    <Name>STTestBuck</Name>
    <Prefix />
    <MaxKeys>1000</MaxKeys>
    <IsTruncated>true</IsTruncated>
    <Contents>
      <Key>question-mark.jpg</Key>
      <LastModified>2011-11-16T11:27:14.000Z</LastModified>
      <ETag>a6740be302e1e0617c843d606c1a470b</ETag>
      <Size>143</Size>
      <Owner>
        <ID>KxSyw67T0wVUvcZoGRMo07FGH0NXAoCpPTF5Q</ID>
        <DisplayName>cloudtest</DisplayName>
      </Owner>
      <StorageClass>STANDARD</StorageClass>
    </Contents>
    ...
    ...
  </ListBucketResponse>
</ListBucketResponse>

Your quick reply will be appreciated. Thank you once again.

Regards, Rajesh A.